Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 1: Overview, Tokenization
Percy Liang's opening Stanford CS336 lecture explains why the course builds language models from scratch under a compute budget, then works through tokenizers: characters, bytes, words, and byte pair encoding, trained, encoded, and decoded step by step.
3 sections ยท 8 lessons
Course outline
Budget and Tokenizer Basics
- Building to Understand, Under a Budget
- The Tokenizer's Two-Way Contract
- Compression Ratio Versus Vocabulary Size
Characters, Bytes, Words
- Code Points Are Not Bytes
- Full Coverage Versus Short Sequences
Byte Pair Encoding
- Training BPE on "the cat in the hat"
- Encoding New Text With Frozen Merges
- Boundaries: Pre-tokenization and Special Tokens