Real-Time Ad Event Pipelines: Lessons from Netflix

Understand how to design scalable event enrichment pipelines using streaming joins, choose the right delivery guarantee per layer, and safely migrate a revenue-critical production system without downtime

4 sections ยท 13 lessons

Course outline

The Problem: Synchronous Writes on the Critical Path

  1. What is the critical path in ad serving?
  2. The Metadata Registry bottleneck at Live scale
  3. Fire-and-forget logging vs. synchronous write-and-wait

Streaming Joins with Apache Flink

  1. How Flink stateful stream processing works
  2. KeyedCoProcessFunction: owning the join at the lowest API level
  3. The 60-minute match window and long-tail recovery
  4. Keeping stateful jobs lean: one job, one responsibility

Cross-Region Routing and Delivery Guarantees

  1. Route the event, not the data
  2. At-least-once vs. exactly-once: choosing the right guarantee per layer
  3. Downstream idempotency: letting each consumer own its guarantee

Safe Production Cutover

  1. Running two pipelines in parallel with shared enrichment logic
  2. Deterministic traffic splitting: avoiding double-counting
  3. Incremental rollout and knowing when to decommission

Start learning with Wondering