Partial Failure Is the Normal Case

When Part of the System Breaks

In April 2011, one network change left about 13% of an AWS zone's EBS volumes stuck.[[2]](https://aws.amazon.com/message/65648/) (EBS, Elastic Block Store, provides AWS's replicated storage volumes, the disks its virtual servers use.) Most machines kept running the whole time.

That is partial failure: some components fail while the rest keep going.

The hard part is uncertainty. A healthy node often cannot tell whether a peer is dead, slow, or just unreachable.

A single server that crashes and restarts cleanly is the easier case. Nothing else keeps running while guessing about its state.

At Scale, Something Is Always Broken

Amazon's Dynamo paper describes tens of thousands of servers and network components, where small and large components fail continuously.[[1]](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)

Datacenter Server Racks (22370909788).jpg) · Carl Lender from Sunrise, USA · CC BY 2.0 · Wikimedia Commons

So the shopping cart had a blunt requirement: accept writes even if disks are failing, network routes are flapping, or data centers are destroyed by tornados.[[1]](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)

At that size, failure is the normal state. Handling it belongs in the main design.

Redundancy helps, but it does not prevent outages by itself. AWS's redundant storage still failed, through its own recovery path.

Your Slowest Call Sets the Pace

One Amazon page could call over 150 services.[[1]](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)

With that much fan-out, the page waits for every call. One slow service slows the whole page, even when the average looks fine.

That is why Dynamo set targets at the 99.9th percentile, for example 300ms for 99.9% of requests at 500 requests per second.[[1]](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)

Averages hide the users stuck in the tail. A slow peer is a partial failure too.

Anatomy of the 2011 EBS Cascade

At 12:47 AM PDT on April 21, 2011, a network change sent EBS traffic onto a lower-capacity backup network.[[2]](https://aws.amazon.com/message/65648/)

Many storage nodes lost contact with their replicas. Each node treated its missing replica as gone.

On reconnection, the nodes all re-mirrored at once.[[2]](https://aws.amazon.com/message/65648/) The cluster exhausted its available capacity, and slow API calls then failed requests in other zones.

Recovery Is Also Load

AWS called it a re-mirroring storm, and it left about 13% of volumes in the zone stuck.[[2]](https://aws.amazon.com/message/65648/)

Each node did something reasonable: it searched the cluster for storage space for a new replica. Together, they put the cluster into the storm, and it exhausted its available capacity.[[2]](https://aws.amazon.com/message/65648/)

Notice that every arrow is a sensible recovery action, yet together they overflow the small spare pool and leave volumes stuck.

Retries behave the same way. Without backoff, every client hits a struggling service at its weakest moment.

AWS's fixes included more spare capacity and backoff.[[2]](https://aws.amazon.com/message/65648/) The next question is how the damage escaped the zone.

A Shared Control Plane Crosses Zones

Availability Zones exist so one zone's trouble stays local. In 2011, it did not.

The control plane served API requests from a regional thread pool: a fixed number of worker threads shared by all zones. When every thread is busy, new requests have nothing to run on.

Slow create-volume calls in the troubled zone filled that pool. API requests for other zones then failed too.[[2]](https://aws.amazon.com/message/65648/)

The zones had their own storage, but they shared one control plane.

A fault travels along any shared dependency. So when you map failure domains, include the management layer along with the servers.

Design for the Uncertain Middle

Even redundancy built for failover fell short: 2.5% of multi-AZ RDS instances (managed databases set up to fail over to another zone) did not fail over automatically, and 0.07% of volumes could not be restored consistently.[[2]](https://aws.amazon.com/message/65648/)

In the EBS recovery path, nodes that could not find space did not back off aggressively enough and kept searching, and everything leaned on one regional control plane.[[2]](https://aws.amazon.com/message/65648/)

Dynamo's cart aims the other way. Through hinted handoff, any healthy node accepts the write, so one failed node does not block the cart.

In your own designs, ask: what do retries do under stress, and what do all zones share?

A distributed system has some failed components while other components continue serving requests. Is this an example of partial failure?

• Yes
• No

Answer: Yes

Yes. Partial failure means that only part of the system fails while the rest keeps running. This differs from a single process crash where the affected system may simply stop and restart without other components making decisions under uncertainty.

Why can a system built from individually reliable, redundant components still experience an outage?

• Redundant components cannot fail at the same time under any circumstances
• Recovery actions and shared dependencies can turn local faults into system-wide overload
• Redundancy makes failure detection unnecessary because another copy exists
• A redundant system always routes every request through one storage node

Answer: Recovery actions and shared dependencies can turn local faults into system-wide overload

Redundancy helps tolerate component failures, but the mechanisms that detect and recover from failures can create new overload. In the EBS incident, many reasonable re-mirroring actions consumed spare capacity, and shared control-plane resources spread the impact.

A storage zone loses contact with many replicas, and all nodes immediately retry replication. The retries consume spare capacity; slow volume-creation calls then fill a regional thread pool used by other zones. Which design change most directly reduces this cascade?

• Retry immediately from every node until replication succeeds
• Use backoff for retries and protect recovery with reserved spare capacity
• Remove replicas so fewer nodes participate in recovery
• Route all zones through the same unrestricted control-plane pool

Answer: Use backoff for retries and protect recovery with reserved spare capacity

Backoff prevents every node or client from retrying at once, reducing synchronized load during recovery. Reserved spare capacity also helps, but backoff directly addresses the simultaneous retry storm described. The shared regional thread pool remains a dependency that should also be isolated or protected.