Durable Execution for AI Agents: Checkpoint, Replay, Recovery

Durable execution is not a retry library and it is not a queue. A queue guarantees that a message is delivered at least once. Durable execution guarantees something stronger and more specific: that a multi-step program resumes at the step boundary it reached, with the same local variables, after the process running it disappears. The unit of durability is the step, not the request.

The mechanism is a persisted event journal plus replay. Every time your agent completes an externally-visible action — an LLM call, a tool invocation, a timer, a human approval — the runtime appends a record of that action and its result to durable storage. If the worker crashes, a new worker loads the journal and re-executes the agent function from the top. This time, when the code reaches a step that already has a journal entry, the runtime does not call out again; it returns the recorded result immediately. Execution fast-forwards through completed work in microseconds and then continues live from the first uncompleted step.

That one mechanism produces the guarantees teams actually want. Persistence: state survives deploys, OOM kills, and spot-instance reclaims. Effectively-once side effects: a tool call that already succeeded is never re-issued during recovery. Suspension: an agent can wait days for a human without holding a process, because the waiting state lives in the journal, not in memory. What it does not give you is protection against a step that is itself non-idempotent and crashes mid-flight — that gap is real, and the idempotency section below is how you close it.

Walk one turn of the refund agent. It reads ticket TCK-4417, asks a model to classify intent, looks up order ORD-9912, calls the payments API to refund $120, waits for a supervisor approval signal, then emails the customer. Six steps, five of which touch the network. In a plain async function, a crash after step four means the money moved and nothing knows it. In a durable function, step four appended ActivityCompleted{id: 4, result: {refund_id: "re_1P2x"}} to the journal before step five began.

Recovery replays the function body from line one. Steps one through four hit their journal entries and return recorded values without any network traffic. Step five — the approval wait — is where the journal ends, so the runtime blocks there for real and the agent continues forward from a consistent state. The refund is not re-issued, because the code path that would issue it returned a cached result instead of calling Stripe.

This is why replay imposes a hard constraint on orchestration code: it must issue the same sequence of steps given the same inputs. Temporal states this plainly — workflow code must make "the same Workflow API calls in the same sequence, given the same input," and it detects violations by comparing newly-generated commands against the recorded event history, raising a non-determinism error on mismatch. If your agent function branches on Date.now() or Math.random(), replay takes a different branch than the original run, the command sequence diverges, and the runtime cannot tell recovery from corruption.

An LLM is the most non-deterministic component you will ever put in a workflow. The same prompt at temperature 0.7 returns different text, different tool arguments, and sometimes a different number of tool calls. If you call the model directly from orchestration code, replay will produce a different plan than the original run and the command sequence will diverge on the first divergent token.

The fix is a boundary, not a setting. The model call goes inside an activity — code that runs outside the replay path. The activity returns its output, that output is journaled, and orchestration code treats it as a fixed input from then on. On replay the model is never called again; the recorded completion is handed back. This is the same trick Temporal uses for SideEffect: record the value once, replay the record.

The subtlety is agent loops. A ReAct-style agent decides its next tool from the model's last output, so the number of iterations is itself non-deterministic. That is fine — because each iteration's model output is journaled, replay follows exactly the branch the original run took. What breaks is deriving loop control from something outside the journal: a Date.now() deadline, a live feature flag read, or a shared counter in Redis. Those must become journaled step results too. For the memory side of this problem, AI Agent Memory: Short-Term vs Long-Term covers what belongs in the journal versus an external store.

The journal protects you between steps. It cannot protect you inside one. If the worker dies after Stripe accepted the refund request but before the activity returned, no journal entry exists — and the runtime, seeing an unfinished activity, will retry it. Without extra work, that retry issues a second $120 refund on ORD-9912.

The standard answer is an idempotency key: a caller-supplied identifier that the downstream service uses to deduplicate. Send Idempotency-Key: refund-ORD-9912-attempt-1 and Stripe returns the original response for any repeat of that key rather than creating a second charge reversal. The key must be stable across retries of the same logical operation and different across genuinely different operations, which means it must be derived from workflow identity — not from crypto.randomUUID() inside the retry path, which produces a fresh key per attempt and defeats the whole mechanism.

For services that offer no idempotency support, you own the deduplication. Write an intent row (refund_intents(key, status)) in your own database inside the same transaction that records the attempt, check it before calling out, and reconcile on startup. That is the same shape as the Transactional Outbox Pattern, applied to outbound side effects instead of published events. The honest framing is the one from Exactly-Once Delivery: Myth vs Reality: you get at-least-once delivery plus deduplication, and the combination behaves like exactly-once effects.

Trigger: a routine deploy rolled workers during a traffic spike. Twenty-three refund agents were mid-flight; four of them were inside the issueRefund activity when SIGTERM arrived and the grace period expired.

Symptom: finance reported four orders — ORD-9912 among them — with two refunds each, six minutes apart. The agent traces looked clean. Every workflow reported success. Nothing errored.

Root mechanism: the activity had a retry policy but no idempotency key. Stripe accepted the first request and returned a response the dying worker never journaled. The runtime saw an activity with no completion event, applied its retry policy exactly as configured, and called Stripe again. Both calls were valid, independent requests. The journal was not wrong — it recorded one completed activity, because from its perspective only one attempt ever finished. The bug was that the effect was not tied to the identity of the attempt.

Evidence and recovery: the diagnostic was the Stripe dashboard, not the agent logs — two re_ ids with distinct request_id values and no shared idempotency key proved the retry theory in under a minute. Recovery was a reconciliation script that grouped refunds by payment intent within a 30-minute window and reversed the later duplicate. Prevention was the key derivation shown above, plus a regression test that kills the worker with SIGKILL between the HTTP call and the return statement and asserts the payment provider recorded exactly one refund.

Pick by the shape of the failure you cannot tolerate, not by the feature matrix. If a repeated side effect would be a customer-visible incident and runs last longer than a request timeout, you want a real engine. If your agent is a three-step retrieval-and-summarize chain with no external mutations, a status column and a cron sweeper is the correct amount of machinery, and adding Temporal buys you an operational burden with no matching risk reduction.

The boundary case that trips teams up is the middle: agents that mutate state but finish in under a minute. Here a plain state machine works, and the AI Workflow State Machines approach is usually enough — provided you still write idempotency keys, because crash-mid-call does not care how short your workflow is. Adopt a durable engine when you start needing suspension across hours, fan-out with partial failure, or a compensation path per step; those are the three things hand-rolled state machines consistently get wrong.

Every option here shares one cost worth naming up front: your orchestration code becomes constrained code. Refactors that reorder steps break running instances, local debugging needs a replay harness, and new engineers need to learn which lines run once and which run on every recovery. Budget for that learning curve the way you would budget for adopting a service mesh.

Build the smallest version of the refund agent and prove the guarantee yourself. Start from any durable runtime you can run locally — Temporal's dev server, DBOS on a local Postgres, or Inngest's dev CLI. Implement four steps against a fake payments service you control: getTicket, classifyIntent (a stub returning a fixed plan), issueRefund, and sendEmail. Have the fake payments service append every request it receives to a file, including the idempotency key, so you can count effects rather than trust logs.

Run it once with ticket TCK-4417 and order ORD-9912. Expected success: the workflow completes, and the payments log contains exactly one line. Now break it on purpose — add a process.exit(1) inside issueRefund immediately after the HTTP call returns but before the function returns its value, and run again. The pass criterion is precise: after the runtime restarts and the workflow completes, the payments log has exactly two request lines with the same idempotency key, and the fake service returned the cached response the second time, so exactly one refund exists.

If you see two distinct refunds, your key is not derived from stable identity — check that it uses the run id and activity id rather than a value generated inside the retry path. If the workflow refuses to resume with a non-determinism error, you put a clock, a random value, or the model call in orchestration code; move it into an activity and rerun.