Three Training Stages

From Base Model to Assistant

How does a neural network become a helpful assistant like ChatGPT? It passes through three training stages in a fixed order.

First, pre-training learns from internet documents. Then supervised fine-tuning (SFT) teaches assistant behavior.

Finally, reinforcement learning (RL) provides problem-solving practice.

Each stage builds on the previous one, and all three are essential to the final product.

Pre-training Builds the Foundation

Pre-training feeds the model internet documents. It runs for months on thousands of GPUs.

Many documents converge on one model supported by thousands of GPUs, revealing why pre-training demands so much computation.

The result is a base model with broad patterns learned from that text.

But a base model mainly simulates internet documents. Given text, it continues in the style of an internet document.

That makes it powerful, but not directly useful as an assistant yet.

SFT Teaches Assistant Behavior

Supervised fine-tuning (SFT) turns the base model into an assistant.

During SFT, the model trains on human-labeled conversations. People provide questions paired with good assistant answers.

These examples teach the pattern of a helpful reply. The model moves from continuing document-style text toward responding to users.

Pre-training supplied broad patterns. SFT changes how the model behaves with that foundation.

RL Provides Problem-Solving Practice

The third stage is reinforcement learning (RL). The model practices solving problems itself.

It tries approaches, checks whether they reach good solutions, and reinforces what worked.

The reinforced strategy can guide the next attempt, so practice forms a repeating cycle. This helps the model discover effective problem-solving strategies.

Think of a student doing practice problems until a reliable method sticks.

The School Learning Analogy

The pipeline mirrors how people learn in school.

First, we read textbooks to build background knowledge. That is pre-training, where internet documents produce a base model.

Then we study worked solutions from experts. That is SFT, where human-labeled conversations teach assistant behavior.

Finally, we do practice problems ourselves. That is RL, where the model’s own attempts discover better problem-solving strategies.

Each stage builds on the previous one.

Where the Computing Goes

The stages differ hugely in computational cost. Pre-training is by far the most expensive.

It takes about 3 months, uses thousands of GPUs, and costs millions of dollars.

Post-training, meaning SFT and RL, lasts hours to days. One example is only about 3 hours.

The cost drops dramatically, yet every stage remains essential. Economics therefore shapes which capabilities teams develop in each stage.

Recognize Each Stage’s Influence

Each stage leaves a different mark on an assistant.

Broad knowledge comes from pre-training. Helpful conversational behavior comes from SFT, while RL develops problem-solving strategies through practice.

The economics matter: pre-training takes about 3 months and costs millions of dollars, while a post-training example takes about 3 hours.

When evaluating a capability or limitation, ask which stage likely shaped it. This separates cheaper post-training changes from changes that depend on costly pre-training.

After pre-training alone, a language model is _____

• a base model that imitates internet documents
• a helpful assistant ready to answer questions

Answer: a base model that imitates internet documents

Pre-training creates a base model that simulates internet documents, continuing text the way a web page might. It is not yet useful as an assistant. Supervised fine-tuning is needed to make it one.

LLM training has three stages: _____ learns from internet documents, _____ learns from human-labeled conversations, and _____ discovers strategies through practice.

Answer: LLM training has three stages: pre-training learns from internet documents, supervised fine-tuning learns from human-labeled conversations, and reinforcement learning discovers strategies through practice.

Each stage has a distinct data source and goal: - Pre-training learns from internet text and produces a base model. - Supervised fine-tuning trains on labeled conversations, which teaches assistant behavior. - Reinforcement learning lets the model practice problems and find good strategies.

LLM training is often compared to how students learn from a textbook. In that comparison, studying worked solutions matches which training stage?

• Supervised fine-tuning
• Pre-training
• Reinforcement learning
• Data collection

Answer: Supervised fine-tuning

The comparison maps each step of learning to a stage: - Reading textbooks matches pre-training. - Studying worked solutions matches supervised fine-tuning, because the model imitates expert-written example responses. - Doing practice problems matches reinforcement learning.