Debugging Production-Scale Spark Pipelines
Be able to design and run parallel, branch-isolated Spark experiments to catch and fix bugs that only appear at full production scale, including data skew, collect_list memory bloat, and AQE broadcast join pitfalls.
4 sections ยท 9 lessons
Course outline
Why Scale-Only Bugs Slip Past Small-Sample Testing
- Data skew and why 1% samples miss it
- When you must test against full, unsampled production data
Building a Parallel Experimentation Loop
- Branch-driven isolation for concurrent experiments
- Using tmux for session persistence on remote dev boxes
- Delegating experiment ops to an AI agent
Spark Memory Internals That Bite at Scale
- collect_list and per-row memory bloat
- How AQE broadcast joins can silently pin executor memory
Turning Tuning Into Lasting Fixes
- Keeping a single experiment log to isolate variables
- Why tuning buys headroom, not a new ceiling