Course I — How Language Models Learn · Module 1 — From Text to a Learning Problem · Unit 2

Tokens, IDs, Vectors, and Tensors

Unit contract

Central question: What representation does a language model actually receive when a human sees a sentence?

Prerequisites: The Model Lifecycle.

Learning outcomes:

• distinguish text, tokens, token IDs, embeddings, vectors, matrices, and tensors;
• read the shapes of a tiny batch and sequence;
• explain why tokenisation is a modelling and systems decision;
• follow one short sentence into the input of a model without treating a token as a word.

Planned sections:

1. Text is not the model's input.
2. Tokenisation and vocabulary IDs.
3. From discrete IDs to learned vectors.
4. Batch, sequence, and feature dimensions.
5. Representation choices and their consequences.

Mathematical bridges: vectors, matrices, dimensions, indexing, and tensor shapes. No matrix multiplication yet.

Candidate evidence or figures: a course-authored trace of one sentence; an optional live tokenizer inspection if it improves the explanation.

Primary sources: to be selected during Spike 2.

Non-goals: deriving tokenisation algorithms or explaining attention.

Lesson

This unit has not been authored. Its contract is the target for Development Spike 2.

Estimated study time: 75 minutes