TD-MPC Explained, With Alexander Soare (Part 2 of 2)

In this video I explain how we train the neural networks of TD-MPC. TD-MPC paper: https://arxiv.org/abs/2203.04955 FOWM paper (this is what's behind the implementation in the LeRobot library): https://arxiv.org/abs/2310.16029 LeRobot code: https://github.com/huggingface/lerobot/blob/main/lerobot/common/policies/tdmpc/modeling_tdmpc.py Many thanks to Nicklas Hansen et. al. for publishing their research and open sourcing their code. Chapters: 0:00 - Listing the neural networks we need to train 04:53 - What a training batch item looks like 06:09 - Forward passes and losses 13:41 - Why the latent state representation does not collapse 14:24 - Understanding TD Learning 23:42 - TD learning intuition in real experiments 26:58 - Optimizing the Q network using the TD error 30:34 - Offline vs online data collection and training loop 36:20 - Wrapping up