Free sample lesson · Deep Dive Into LLMs
Three Training Stages
How does a neural network learn to be a helpful assistant?
Lesson 1 of the Wondering course Deep Dive Into LLMs
Three Stages, One Assistant
A helpful assistant is not produced by a single training run.
Large language models move through three distinct stages: pre-training on internet documents, supervised fine-tuning on assistant conversations, and reinforcement learning on problem solving.
Each stage builds on the one before it. The order is fixed, and the capability a user finally touches is shaped by all three.
- Pre-training: The model learns from a large collection of internet documents.
- Fine-tuning: The model learns to behave like an assistant in conversations.
- Reinforcement learning: The model practices solving problems.
Like Going Through School
The pipeline mirrors how people are educated.
First you read textbooks and absorb background knowledge. Then you study worked solutions to see how an expert responds. Finally you do practice problems and find out what actually works.
Training follows the same three moves, in the same order.
Pre-training Is The Big Spend
Pre-training is by far the most computationally expensive stage.
It runs for months on thousands of GPUs, taking roughly 3 months and costing millions of dollars.
CSIRO ScienceImage 11313 The CSIRO GPU cluster at the data centre · division, CSIRO · CC BY 3.0 · Wikimedia Commons
For a product manager, this is the line item that cannot be iterated on casually. One run is a budget cycle, not a sprint.
The Base Model Is Not A Product
Pre-training produces a base model. It simulates internet documents.
That is a real capability, and it is still not directly useful. Ask it a question and it may continue the text rather than answer you.
Millions of dollars have been spent, and there is no assistant yet.
Fine-tuning Creates The Assistant
Supervised fine-tuning, or SFT, converts the base model into an assistant.
The method is straightforward: train on human-labeled conversations that show what a good assistant reply looks like.
The knowledge does not come from this stage. The behavior does.
- Base model
- After SFT
Reinforcement Learning Is Practice
Reinforcement learning lets the model practice problem solving.
Instead of copying a labeled answer, the model attempts a problem, the outcome is checked, the strategies that worked are reinforced, and it tries again.
Repeated many times, this practice loop lets the model discover strong problem-solving strategies rather than imitate them.
- Attempt problem: The model produces its own solution rather than copying one.
- Check outcome: The result is evaluated.
- Reinforce what worked: Successful strategies are strengthened.
- Try again: The model practices once more with what it learned.
Post-training Is Cheap
Fine-tuning and reinforcement learning together are called post-training.
The contrast in cost is dramatic. Pre-training takes around 3 months and millions of dollars. Post-training runs in hours to days, roughly 3 hours, and is far cheaper.
Same pipeline, radically different economics.
- Pre-training
- Post-training
Which Capabilities Get Built
Cost shapes the roadmap. Because post-training is cheap, capabilities that can be added through fine-tuning or practice get developed quickly and often.
Anything that requires a fresh pre-training run competes against millions of dollars and about 3 months of cluster time.
So the cheap stage quietly decides much of what users actually see.
- New capability: A capability is requested for the assistant.
- Post-training enough?: Can fine-tuning or practice deliver it?
- Post-train it: Cheap enough to develop quickly and often.
- New pre-training run: Costs millions of dollars and about 3 months of cluster time.
- Users see it
Every Stage Is Essential
The computational cost falls sharply at each stage, and every stage still matters.
Pre-training supplies the knowledge foundation. Fine-tuning supplies assistant behavior. Reinforcement learning supplies practiced problem solving.
Remove the bottom and nothing above it stands. Remove the top and the product feels less capable.
- Reinforcement learning: Practiced problem solving on top of assistant behavior.
- Supervised fine-tuning: Assistant behavior learned from human-labeled conversations.
- Pre-training: The knowledge foundation built from internet documents.
Each level rests on the one below.
Check yourself
Does pre-training alone produce a finished assistant that reliably answers user questions?
- Yes
- No
Answer: No
No. Pre-training produces a base model that simulates internet documents and may continue text instead of answering. Supervised fine-tuning is needed to create assistant behavior.
Training uses _____ to build the base model, _____ to teach assistant behavior, and _____ to develop strategies that work.
Answer: Training uses internet documents to build the base model, human-labeled conversations to teach assistant behavior, and problem-solving practice to develop strategies that work.
The three stages use different inputs or activities. Pre-training uses internet documents, supervised fine-tuning uses human-labeled conversations, and reinforcement learning uses repeated problem-solving practice.
A model attempts a math problem, receives feedback on whether its solution worked, and repeats the process. Which training stage best matches this activity?
- Pre-training
- Supervised fine-tuning
- Reinforcement learning
- Document collection
Answer: Reinforcement learning
Reinforcement learning involves attempting problems, checking outcomes, reinforcing successful strategies, and trying again. Supervised fine-tuning instead teaches from labeled example conversations.
Keep going
This is lesson 1 of 20 in Deep Dive Into LLMs. Next up:
- Internet Data Curation
- Tokenization Fundamentals
- Neural Network Training
- Base Models Explained
- Text Generation Process
Wondering turns any topic into short lessons you can read, watch, or listen to, with practice questions and an AI tutor.