Three Training Stages

Three Stages, One Pipeline

Modern LLM assistants are not trained once. They pass through three sequential stages, each reshaping the same weights.

Pre-training learns from internet documents. Supervised fine-tuning learns assistant behavior. Reinforcement learning practices problem-solving.

Each stage depends on the checkpoint produced before it. The order is not optional.

The Education Analogy

The pipeline mirrors how humans are educated.

First we read textbooks to absorb broad knowledge. Then we study worked solutions to see how experts respond. Finally we do practice problems.

Pre-training is the textbook. Supervised fine-tuning is the worked solution. Reinforcement learning is the practice set.

Pre-training And The Base Model

Pre-training runs for months on thousands of GPUs and costs millions of dollars.

What comes out is a base model. It simulates internet documents.

That is a powerful density estimator, but it is not directly useful. Ask it a question and it may continue the text rather than answer it.

From Base Model To Assistant

Supervised fine-tuning converts the base model into an assistant.

The training data changes character. Instead of raw internet documents, the model sees human-labeled conversations.

The weights barely move compared to pre-training, but the behavior shifts sharply. The model now answers instead of continuing text.

Reinforcement Learning As Practice

Supervised fine-tuning shows the model what a good answer looks like. It does not show the model how to find one.

Reinforcement learning fills that gap. The model practices problem-solving and discovers optimal strategies through repeated attempts.

This is the stage where the model learns strategy rather than imitating a labeler.

The Cost Cliff

The compute budget collapses across the pipeline.

Pre-training takes roughly 3 months and costs millions of dollars. Post-training can finish in roughly 3 hours.

That is a difference of about three orders of magnitude in wall-clock time. Post-training is measured in hours to days, not months.

Why The Economics Matter

Cost asymmetry shapes which capabilities get built.

Almost nobody re-runs pre-training, so the knowledge baked into the base model is largely frozen. Post-training is cheap, so behavior gets iterated constantly.

Every stage remains essential. But the cheap stages are the ones that get tuned, and that explains much of how these assistants behave.

Supervised fine-tuning primarily teaches an AI model to behave like an assistant rather than merely continue internet text.

• True
• False

Answer: True

Correct. Supervised fine-tuning uses human-labeled conversations to shift the model toward answering users and following assistant behavior. Reinforcement learning instead provides repeated problem-solving practice.

The training pipeline uses _____ to create a base model, _____ to shape assistant behavior, and _____ to develop strategies.

Answer: The training pipeline uses internet documents to create a base model, human-labeled conversations to shape assistant behavior, and repeated problem-solving attempts to develop strategies.

Each blank identifies the distinctive input or activity of one stage: pre-training uses internet documents, supervised fine-tuning uses human-labeled conversations, and reinforcement learning uses repeated attempts.

A newly created base model is asked, “What is the capital of France?” Which behavior is most characteristic before supervised fine-tuning?

• It may continue the prompt as if writing a document
• It reliably follows conversational instructions
• It immediately discovers an optimal solution through practice
• It contains no knowledge from internet data

Answer: It may continue the prompt as if writing a document

The base model simulates internet documents, so it may continue text that resembles the prompt instead of reliably answering as an assistant. Supervised fine-tuning shifts this behavior toward direct responses.