Cascading failure 如何發生

One Slow Service, Many Victims

A cascading failure starts small. One service deep in your system slows down or stops responding.

Soon the services that call it slow down too. Then the services that call them start failing.

The trouble spreads upstream, toward the user, one caller at a time.

Notice that the red trouble starts at the last service and moves backward through each caller until it reaches the user.

The key question for this lesson: how does a problem in someone else's service eat up your threads and connections?

Slow Is Worse Than Down

It sounds backward, but a slow downstream service often does more damage than one that is fully down.

When a service is down, calls fail fast. The caller gets an error right away and moves on.

When a service is slow, every call just waits. The caller holds its resources the whole time.

Waiting is what drags the caller down.

Threads Stuck Waiting

Most services handle each request with a thread from a fixed-size thread pool.

When a thread calls a slow downstream service, it blocks. It cannot serve anyone else until the reply comes back.

New requests keep arriving. Each one grabs another thread, and that thread gets stuck too.

Eventually no free threads remain, and new requests queue up or get rejected.

Connections Run Out Too

Threads are not the only thing being held. Each call to a downstream service also uses a connection, usually from a connection pool.

A slow reply keeps that connection busy. Other calls must wait for one to free up.

So the caller runs out of connections even though its own code is fine.

Both pools share one pattern: slow replies hold resources longer than normal.

Retries Pour Fuel on It

When calls time out, callers often retry. That feels safe, but it adds even more requests to a service that is already struggling.

More load makes the downstream service slower.

Slower replies cause more timeouts, which trigger more retries. The problem feeds itself in a loop.

The Healthy Service Falls

Here is the surprising part. Once your threads and connections are all stuck, your service stops answering too.

To your own callers, you now look slow or down, even though nothing in your code broke.

Their threads start waiting on you. The same drain repeats one level up, and the failure keeps climbing toward the user.

Spotting and Stopping It

You can recognize a cascade by its signals: threads busy but doing no work, empty connection pools, and rising timeouts on calls to one dependency.

The fix is to stop waiting forever.

Set short timeouts, keep pools bounded, and fail fast when a dependency is clearly unhealthy. Be careful with retries.

In plain words: never let someone else's slowness hold all your resources.

In a cascading failure, the trouble spreads _____, from a slow deep service toward the user.

• upstream
• downstream

Answer: upstream

The failure moves upstream: callers of the slow service slow down, then their callers fail, and so on toward the user, one caller at a time.

When a downstream service is slow, each request's _____ blocks and cannot serve anyone else. Each call also keeps a _____ busy. Timed-out calls often trigger a _____, which adds even more load.

Answer: When a downstream service is slow, each request's thread blocks and cannot serve anyone else. Each call also keeps a connection busy. Timed-out calls often trigger a retry, which adds even more load.

A slow dependency holds two pooled resources longer than normal: threads (from the thread pool) and connections (from the connection pool). Retries after timeouts add load to the struggling service, making things worse.

An order service handles requests with a fixed pool of 200 threads. Its payment dependency starts taking 30 seconds to reply instead of 50 ms, while traffic stays the same. What happens to the order service?

• Threads pile up waiting until none are free, so new requests queue or get rejected
• Threads receive errors immediately and are released, so capacity is unaffected
• Only payment requests are affected; other order-service requests are served normally
• The thread pool automatically grows to absorb the extra waiting requests

Answer: Threads pile up waiting until none are free, so new requests queue or get rejected

Each thread calling the slow payment service blocks until the reply arrives. New requests keep grabbing threads, which also get stuck. Eventually no free threads remain, so new requests queue up or get rejected, even though the order service's own code is fine.