How Large Language Models (LLMs) Work

This technical guide explains how large language models work through transformer architecture and self-attention mechanisms that process relationships between all elements in a sequence simultaneously. LLMs consist of hundreds of millions or billions of parameters pre-trained on massive text corpora, enabling them to understand word associations regardless of position in a sentence. As of 2024, multimodal LLMs have become mainstream, with major providers now processing images, audio, and video alongside text.

10 sections ยท 27 lessons

Course outline

Foundational Concepts

  1. Text to Numbers
  2. Layered Processing
  3. Tokenization
  4. Embeddings
  5. Foundational Concepts

Sequential Processing Problems

  1. Sequential Limitations
  2. Sequential Processing Problems

Transformer Architecture

  1. Parallel Processing
  2. Query-Key-Value
  3. Attention Weights
  4. Transformer Architecture

Self-Attention Mechanics

  1. Self-Attention
  2. Causal Masking
  3. Multi-Head Attention
  4. Self-Attention Mechanics

Model Architectures

  1. Architecture Types
  2. Model Scale
  3. Model Architectures

Pre-Training Process

  1. Pre-Training
  2. Next-Word Prediction
  3. Pre-Training Process

Fine-Tuning and Alignment

  1. Fine-Tuning: Specializing Pre-Trained Models
  2. RLHF Overview
  3. RLHF Stages
  4. Fine-Tuning and Alignment

Text Generation

  1. Autoregressive Generation
  2. Softmax Transformation
  3. Temperature: Controlling Output Randomness
  4. Sampling Strategies
  5. Text Generation

Generation Control

  1. Creativity vs Precision
  2. Generation Control

Model Limitations

  1. Hallucinations
  2. Mitigating Hallucinations
  3. Context Windows
  4. Context Management
  5. Model Limitations

Start learning with Wondering