I Heart Logs (Jay Kreps): The Log as the Backbone of Distributed Data

Explain, chapter by chapter from Jay Kreps' 'I Heart Logs', why the append-only log is the unifying abstraction behind databases, consensus/replication, data integration, and stream processing, and confidently discuss offsets, CDC, log compaction, partitioning, and the Kappa-style alternative to Lambda in senior/staff distributed-systems interviews, connecting it to hands-on experience building a Rust CDC change-feed consumer.

4 sections · 21 lessons

Course outline

Ch 1: Introduction — What Is a Log?

  1. The log as a data structure
  2. Logs in databases
  3. Logs in distributed systems: state machine replication
  4. Logs and consensus
  5. Changelog 101: tables and events are dual

Ch 2: Data Integration

  1. Why data integration is hard
  2. Log-structured data flow
  3. Lessons from LinkedIn
  4. ETL, the data warehouse, and organizational scalability
  5. Decoupling systems
  6. Scaling a log

Ch 3: Logs and Real-Time Stream Processing

  1. Stream processing as continuous computation
  2. Data flow graphs
  3. Logs and stream processing: multi-subscriber, ordered, buffered
  4. Lambda architecture: the good and the bad
  5. The alternative: reprocessing from the log
  6. Stateful real-time processing
  7. Log compaction

Ch 4: Building Data Systems with Logs

  1. Unbundling the database
  2. The place of the log in system architecture
  3. Conclusion and interview synthesis

Start learning with Wondering