Crash Timelines

The Crash Timeline Window

Every distributed worker has a dangerous gap: the interval between a local state change and a remote side effect.

In an agent, that means committing a task progress row, then calling an LLM API or a payment gateway.

Notice that the vulnerable region is the space between the local commit and the remote call, not either operation itself.

The crash timeline is the ordering of those two operations relative to process death.

Pinpointing where the process died tells you exactly which recovery is safe.

Two Failure Orderings

There are only two orderings, and each fails differently.

Commit first, crash before the network call: local state claims progress that never happened remotely. The system enters a divergent state.

Call first, crash before the commit: the side effect already landed, and the retry replays it. You get a duplicate side effect.

Neither ordering is safe on its own.

Why Dual Writes Diverge

The root cause is a dual write: two independent systems, no shared transaction.

Your database commit and the payment gateway call cannot be wrapped in one atomic boundary. There is no rollback for a charge that already cleared.

So the fix is not stronger locking. It is coupling state persistence and side-effect triggering as a single logical unit.

That coupling is what atomic state transitions mean in an agent workflow.

Locating The Commit Point

Every crash question reduces to one thing: where is the explicit commit point?

Before the effect, after the effect, or captured in a log that survives the process?

Making that point explicit turns an unbounded set of failure stories into a small, decidable set.

Event-Sourced History

Durable execution frameworks close the gap with an event-sourced history or a transaction log.

Each completed step is appended before the next one runs. On restart, the runtime replays that log instead of guessing.

Recovery restores the instruction pointer and local variables exactly where they left off.

The side effect is no longer a bet against process lifetime.

Runtime Over Retry Loops

For Python and Rust engineers, this is a shift in who owns durability.

Manual try/except retry loops encode recovery in application code, and they only remember what you thought to persist.

Durable execution moves persistence of the call stack into the runtime.

The difference is not effort. It is whether recovery survives a process that no longer exists.

Idempotent Recovery

Knowing the exact moment of failure is what makes recovery idempotent.

If the log says the payment call completed, the retry skips it instead of charging twice. If the log stops before it, the call runs once.

Retries stay safe because replay is decided by durable history, not by a hopeful guess about what already happened.

The Stakes For Agents

Agents raise the cost of getting this wrong.

A double-spend against a payment gateway is unrecoverable. A lost reasoning chain wastes a long-running sequence of LLM calls that must be rebuilt from nothing.

Long-running work spends more wall-clock time inside the crash window, so infrastructure instability is a matter of when, not if.

Design Checklist

Reduce this to habits you can apply on the next service you write.

Name the commit point for every step that touches an external system. Assume the process dies immediately after it.

Then ask what replay does. If replay cannot decide, the step is not yet durable.

The crash timeline is the interval between a local state change and a remote side effect.

• True
• False

Answer: True

Correct. It covers the dangerous gap in which a process may die between updating local state and triggering an external action.

If a worker commits progress before calling a payment gateway, a crash can leave a _____; if it calls the gateway before committing, a retry can cause a _____.

Answer: If a worker commits progress before calling a payment gateway, a crash can leave a divergent state; if it calls the gateway before committing, a retry can cause a duplicate side effect.

A commit-before-call crash records progress that never occurred remotely, producing divergence. A call-before-commit crash may repeat an already completed external action.

Why can a database commit and a payment gateway call not normally be made one atomic transaction?

• The systems have no shared transaction boundary or common rollback
• Database locks are always released before payment requests finish
• Payment gateways cannot receive requests from worker processes
• Distributed workers cannot store progress in databases

Answer: The systems have no shared transaction boundary or common rollback

They are independent systems without a shared transaction boundary. A database rollback cannot undo a payment that has already cleared, so stronger database locking does not solve the dual-write gap.