The Problem with Disk-Based MapReduce

How MapReduce Runs a Job

Hadoop MapReduce splits work into two phases. The map step processes chunks of input, and the reduce step combines the results.

Here is the catch: between stages, MapReduce writes its intermediate results to disk.

Every stage starts by reading from disk and ends by writing back to it. That one rule shapes almost everything about its speed.

Chaining Stages Multiplies the Cost

Real tasks rarely fit in one map and one reduce. Cleaning, joining, and summarizing data often needs several jobs in a row.

In MapReduce, each job hands its output to the next through disk.

So a five-stage pipeline means five rounds of writing and reading back. The computation might be quick, but the data keeps taking the slow road between steps.

Notice that data never flows straight from one stage to the next; it always detours down to disk and back up.

Why Disk Is the Bottleneck

Reading and writing disk is far slower than working with data already in memory.

When a stage finishes, its results do not stay ready for the next stage. They get saved, then loaded again from scratch.

For a single pass over data, this cost is acceptable. For work that touches the same data again and again, the waiting adds up fast.

Iterative Machine Learning Suffers Most

Many machine learning algorithms are iterative. They pass over the same training data many times, improving a model a little on each pass.

In MapReduce, each pass is its own job.

That means every iteration reloads the same data from disk and writes the updated results back. Dozens of passes mean dozens of full disk round trips for data that never changed.

Joins and Ad-Hoc Queries Too

Repeated joins combine tables in several steps. Each join becomes another stage, with another write to disk and another read back.

Ad-hoc queries are questions an analyst asks on the spot, then tweaks and asks again.

An analyst wants answers in seconds. MapReduce reloads data from disk for every new query, so each small change means another slow wait.

Spotting the Pattern in Your Work

Ask one question about any job: does it reuse data across many stages or passes?

If yes, disk-based MapReduce will spend much of its time saving and reloading instead of computing.

Single-pass batch jobs fit it fine. Iterative, multi-step, or interactive work is where it struggles.

Between stages of a job, Hadoop MapReduce stores intermediate results _____.

• on disk
• in memory

Answer: on disk

MapReduce writes intermediate results to disk at the end of each stage, and the next stage reads them back. This rule shapes almost everything about its speed.

In MapReduce, the _____ step processes chunks of input, and the _____ step combines the results. Each stage then saves its output to _____ before the next stage begins.

Answer: In MapReduce, the map step processes chunks of input, and the reduce step combines the results. Each stage then saves its output to disk before the next stage begins.

Map handles input chunks, reduce combines their results, and intermediate output goes to disk between stages, which is the root of MapReduce's slowdown.

A data pipeline in MapReduce has five chained jobs: cleaning, two joins, aggregation, and summarizing. Each job's computation is quick, yet the whole pipeline is slow. What is the most likely reason?

• Each job writes its output to disk and the next job reads it back
• The reduce step must wait for all five map steps to finish first
• Joins require more CPU than the cluster can provide
• Intermediate results are held in memory until the cluster runs out

Answer: Each job writes its output to disk and the next job reads it back

Each job hands its output to the next through disk, so five stages mean five rounds of writing and reading back. The computation is fast, but the data keeps taking the slow road between steps.