Three Training Stages

The Foundation of AI Training

Training a large language model is a three-stage process. It is similar to how a person goes through different levels of education.

First, the model must learn the basics of language. This happens during pre-training.

In this stage, the model reads massive amounts of internet data. It learns to predict the next word in a sentence. This creates a base model that understands patterns but cannot follow instructions yet.

The High Cost of Pre-training

Pre-training is the most resource-heavy phase. It requires thousands of specialized computers working together for a long time.

This stage typically takes about 3 months to complete. It is also very expensive, often costing millions of dollars.

Because of these high costs, only a few large companies can build base models from scratch. Most developers start with an existing base model instead.

Learning to be an Assistant

A base model is like a library of facts. To make it helpful, it needs Supervised Fine-Tuning (SFT).

During SFT, humans provide examples of good conversations. The model learns how to answer questions and follow specific prompts.

This stage is much faster than pre-training. It usually takes only about 3 hours to complete. It transforms a raw document simulator into a useful assistant.

Mastering Problem Solving

The final stage is Reinforcement Learning (RL). This is where the model practices solving difficult problems.

Think of this as a student doing practice exams. The model tries different strategies and receives feedback on which answers are best.

This stage helps the model discover the most efficient ways to think. It ensures the AI is not just repeating data but actually solving the task at hand.

Lesson Wrap-Up

Pre-training an AI model requires _____ resources and time compared to post-training.

• significantly more
• significantly less

Answer: significantly more

Pre-training takes months and millions of dollars, while post-training stages like fine-tuning take far less time and money.

In model development, _____ teaches assistant behavior using dialogue examples, while _____ helps the model practice solving complex tasks using feedback.

Answer: In model development, Supervised Fine-Tuning teaches assistant behavior using dialogue examples, while Reinforcement Learning helps the model practice solving complex tasks using feedback.

SFT uses direct human conversation examples to guide behavior, whereas RL lets the model practice strategies and refine them based on feedback.

What is the primary function of the Reinforcement Learning (RL) phase in LLM training?

• It lets the model practice solving problems and refine strategies through feedback.
• It scans billions of web pages to build an initial vocabulary.
• It reduces the financial cost of purchasing hardware for base model creation.
• It converts audio recordings into written text scripts.

Answer: It lets the model practice solving problems and refine strategies through feedback.

Reinforcement Learning allows the AI to practice problem solving like a student taking practice exams, learning which thinking strategies produce the best results.