Course III — Training Systems at Scale

Central question

What must happen across code, accelerators, networks, storage, schedulers, and people for one large training campaign to make useful progress?

Module 1 — From One Accelerator to a Cluster

1. The Accelerator Execution Model
2. Compute, Memory, and Movement
3. Interconnects and Collectives

Module 2 — Parallelising the Model and Training State

1. Data Parallelism and ZeRO
2. Tensor, Pipeline, and Context Parallelism
3. Expert Parallelism and Mixture of Experts
4. Choosing a Parallelism Topology

Module 3 — Orchestration, Reliability, and Control

1. Launching a Distributed Job
2. Checkpointing and Recovery
3. Determinism and Correctness
4. Observability, MFU, and Goodput

Module 4 — The Organisation Around the Run

1. From Research Proposal to Training Campaign
2. Team Topologies and Responsibilities
3. Inference and Deployment as a Different System
4. Architecture Dossier

Course synthesis

Explain a frontier training campaign as both a distributed system and an organisational control loop.