免費試讀課程單元 · LLM Architecture Deep Dive

Three Training Stages

How does a neural network learn to be a helpful assistant?

Wondering 課程《LLM Architecture Deep Dive》的第 1 個課程單元

What You Will Learn

The Three Training Stages of LLMs

This lesson reveals the fundamental architecture behind AI assistants like ChatGPT. Understanding these stages helps you grasp why these models behave the way they do and what their limitations might be.

The Foundation of AI Training

Large language models are not born as helpful assistants. They follow a three-stage pipeline that mirrors human education.

First, models read textbooks to learn language. Then, they study worked solutions to learn behavior. Finally, they do practice problems to master logic.

The LLM Training Pipeline
  • Pre-training: Reads massive internet text to learn language patterns and form a base model.
  • Fine-Tuning: Studies worked dialogue solutions to learn helpful assistant behavior.
  • Reinforcement Learning: Practices problem-solving to master logic and reasoning.

This process transforms a raw text predictor into a reliable tool for software developers.

Stage One: Pre-training

Pre-training is the most intensive phase. The model learns by reading massive amounts of internet data to understand patterns.

This stage creates a base model. It can simulate internet documents but cannot follow instructions yet.

Pre-training is a massive investment. It typically takes 3 months to complete and requires thousands of GPUs.

Pre-training Resource Requirements

The Economics of Training

The cost of training drops significantly as the model moves through the pipeline.

Pre-training costs millions of dollars and takes months. In contrast, post-training is much faster and cheaper.

Consider these typical timelines: Pre-training takes 3 months. Supervised fine-tuning takes only 3 hours. This allows developers to iterate quickly on specific behaviors.

Pre-training vs Post-training Timeline

Stage Two: Fine-Tuning

Supervised Fine-Tuning (SFT) teaches the model to act as an assistant. Humans provide labeled conversations to show the model how to respond.

This stage converts the raw base model into a helpful chat interface.

It is much shorter than pre-training. While pre-training builds general knowledge, SFT focuses on the specific format of a helpful dialogue.

Training Stages Timeline

Stage Three: Reinforcement Learning

Reinforcement Learning (RL) is the final polish. It allows the model to discover optimal problem-solving strategies through practice.

Think of this as a student doing practice exams. The model tries different approaches and learns which ones yield the best results.

This stage is essential for complex reasoning. It ensures the assistant is not just repeating text but actually solving the user's problem.

How Reinforcement Learning Works
  • Generate Strategy: The model attempts different approaches to solve a user's problem.
  • Test & Evaluate: It tests solutions against expected outcomes to check performance.
  • Refine & Optimize: It learns which approaches yield the best results for complex reasoning.

Putting It All Together

Tracing the Training Pipeline in Software Development
  • Pre-training Foundation: Recognize that the base model already understands programming languages from reading open-source codebases.
  • Fine-Tuning Adaptation: Apply SFT principles when you prompt or fine-tune the model to follow your team's specific API documentation format.
  • RL Optimization: Use RL-aligned prompting or reasoning loops when building coding agents that need to iteratively debug their own outputs.

自我檢測

A _____ is the raw result of the pre-training stage and cannot follow instructions yet.

  • base model
  • fine-tuned model

答案: base model

Pre-training creates a base model that understands language patterns but hasn't been taught to act as an assistant.

In the training pipeline, _____ takes months and costs millions, while _____ is much faster and cheaper for developers to iterate on.

答案: In the training pipeline, Pre-training takes months and costs millions, while Fine-tuning is much faster and cheaper for developers to iterate on.

Pre-training is the most resource-intensive phase, whereas post-training stages like fine-tuning are economically efficient for specific behaviors.

Which stage is specifically designed to help a model discover optimal strategies for complex reasoning through practice?

  • Pre-training
  • Supervised Fine-Tuning
  • Reinforcement Learning

答案: Reinforcement Learning

Reinforcement Learning (RL) acts like a student doing practice exams to find the best way to solve problems, rather than just following examples.

繼續學習

這是《LLM Architecture Deep Dive》共 20 個課程單元中的第 1 個。接下來：

  1. Internet Data Curation
  2. Tokenization Fundamentals
  3. Neural Network Training
  4. Base Models Explained
  5. Text Generation Process

免費開始這門課程

Wondering 把任何主題變成可以閱讀、觀看或收聽的短課程單元，並附有練習題和 AI 家教。