Building to Understand, Under a Budget

Why Build a Model Yourself?

In 2016, a researcher trained their own model. By 2018, they downloaded BERT and fine-tuned it. Today, many simply prompt a model through an API.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

Each step hides more of the machine. That feels convenient until something behaves strangely.

The lecture's point: abstractions are leaky. What happens below the API still shapes what you get, so the course follows understanding via building.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

Building has a catch, though. It costs compute, and compute is limited.

The Pipeline From Text to Loss

Every language model you build follows the same path.

Raw text goes into a tokenizer, which turns it into integer IDs. A model predicts the next ID.

A training loop then adjusts the model to minimize loss on those predictions.

This lecture covers the first stage, tokenization. The course continues through five units, from basics to alignment.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

Why start with the tokenizer? It decides how long every sequence the model sees will be.

Small Models, Big Questions

Training a frontier-scale model is out of reach for a class.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

So you build small, under 1B parameters. The lecture warns these models may not be representative, because some behaviors only emerge with scale.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

That raises a fair worry.

If a small model can mislead you about large ones, what does building it actually teach? The answer depends on which kind of knowledge you want.

Mechanics, Mindset, Intuitions

You implement cross-entropy loss for a small model. Does that skill carry over to a frontier one? Yes.

The lecture sorts knowledge into three kinds. Mechanics (how things work) and mindset (squeezing the most out of hardware, taking scaling seriously) transfer.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

Intuitions about which data and modeling decisions work only partially transfer.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

The trap: the code is the same, so every finding feels portable. The best data mix at 100M parameters may not hold at frontier scale.

Reading the Bitter Lesson Right

The lecture brings up an idea called the bitter lesson, a claim about what drives progress in AI. It weighs two things. Scale means making training bigger: more data and more compute. Algorithms means the methods and design choices themselves, such as how the model and training are built.

The bitter lesson is often read as: scale is all that matters, algorithms do not. The lecture calls that the wrong interpretation.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

The right one: algorithms that scale are what matter.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

The lecture sums this up in a slogan built from three words. Accuracy is how good the final model is. Resources are what you spend: data plus hardware (compute, memory, communication bandwidth).[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json) Efficiency is how much model quality you get out of each unit of those resources. The "x" just says the two work together: more of either helps.

The lecture's slogan is accuracy = efficiency x resources.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json) It is a way of thinking, not a formula; never plug numbers into it.

With resources fixed, efficiency is the lever left. So the course asks for the best model under a fixed compute and data budget.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

The Tokenizer Is a Budget Choice

Two teams get the same compute and the same data. What separates their results? Efficiency.

The lecture states it plainly: "Today, we are compute-constrained."[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json) In other words, the amount of computation you can afford is the limit, so wasted computation directly means a weaker model.

One tokenizer choice is to skip grouping altogether. A computer stores text as raw bytes, small numbers where each character takes one or more bytes. You could hand those bytes straight to the model, one byte per ID. That is elegant: no vocabulary to design, and any text works. But each sequence becomes very long, and the model spends computation on every position. That is what compute-inefficient means here.

Feeding a model raw bytes is elegant, but compute-inefficient with today's architectures.[[1]](https://cs336.stanford.edu/lectures/var/traces/lecture_01.json)

Both rows carry the same text, but the shorter row of tokens leaves more of a fixed compute budget for the model itself.

Longer sequences than necessary waste compute. The team whose tokenizer shortens sequences gets more model out of the same budget.

That makes the tokenizer the first efficiency decision in the whole pipeline.

Implementing cross-entropy loss for a small model is _____.

• mechanics that should transfer to a frontier model
• an intuition that may transfer only partially to a frontier model

Answer: mechanics that should transfer to a frontier model

Implementing how the loss works is mechanics, and the lecture says mechanics transfer to frontier models. The result of a particular data or modeling choice would instead be an intuition with only partial transfer.

In a language-modeling pipeline, raw text first goes to a _____, which produces integer IDs. A _____ predicts the next ID, and a _____ adjusts it to minimize loss.

Answer: In a language-modeling pipeline, raw text first goes to a tokenizer, which produces integer IDs. A model predicts the next ID, and a training loop adjusts it to minimize loss.

The tokenizer converts text into IDs, the model predicts the next ID, and the training loop updates the model to reduce prediction loss.

A team discovers that one data mix works best for a 100-million-parameter model. Which classification and transfer prediction is most accurate?

• Mechanics; it should transfer unchanged to frontier-scale models.
• Mindset; it should transfer unchanged to frontier-scale models.
• Intuition; it may transfer only partially to frontier-scale models.
• Mechanics; it should transfer only partially because larger models use different code.

Answer: Intuition; it may transfer only partially to frontier-scale models.

A preferred data mix is an intuition about which modeling choice works, so it may transfer only partially to frontier-scale models. It is not mechanics, even if the same code can test it.