Course I — How Language Models Learn · Module 1 — From Text to a Learning Problem · Unit 3

Next-Token Prediction

Unit contract

Central question: How does a piece of text become a numerical training signal without anyone labelling its meaning by hand?

Prerequisites: Tokens, IDs, Vectors, and Tensors.

Learning outcomes:

• construct context-target pairs from a token sequence;
• distinguish logits, probabilities, the observed target, and a loss value;
• explain teacher forcing at an intuitive level;
• describe what the loss does and does not tell the model;
• connect next-token prediction to the later need for gradients.

Planned sections:

1. Creating supervision by shifting the sequence.
2. Raw model scores and candidate next tokens.
3. Turning logits into a probability distribution.
4. Measuring surprise at the observed continuation.
5. From one error number to the question of responsibility.

Mathematical bridges: functions, exponentials, logarithms, probability distributions, and negative log-likelihood, introduced through a deliberately small numerical example.

Candidate evidence or figures: a course-authored probability table and loss trace. No frontier-paper figure is required.

Primary sources: to be selected during Spike 3.

Non-goals: fully deriving softmax, performing backpropagation, or explaining why all high-level capabilities emerge.

Lesson

This unit has not been authored. Its contract is the target for Development Spike 3.

Estimated study time: 90 minutes