Durable Agents: Checkpointing, Resumption, and Why Long Tasks Need a Workflow Engine

An agent run that has been going for five hours dies during a routine deploy, and every one of its 240 tool calls evaporates with the process. Nothing about the model was wrong. The plan was sound, the tool results were good, and a human had already approved the risky step at hour two. The run is gone anyway, because all of that state lived in a Python process's memory and a growing context window. When the container restarted, the agent's entire working life restarted with it.

This failure mode barely existed two years ago. Agent tasks were request-shaped - a few seconds of tool calls, then a response. In 2026 they are job-shaped. Deep research runs, codebase migrations, multi-system reconciliations, and compliance reviews now run for hours in the background, and async agent workflows have to be designed to survive failures rather than hope to avoid them. Over a multi-hour window, something will fail. A rate limit, an OOM kill, a deploy, a flaky API. The question is whether that failure costs you one step or the whole run.

The infrastructure answer is older than agents. Distributed systems solved long-running, failure-prone jobs with durable workflow engines, and those engines are now being retrofitted under agent loops. This post covers what to persist, why idempotency has to come before checkpointing, and how to pick among the engines doing this work today.

An Agent Is a Distributed Workflow With a Stochastic Step

The mental model that makes durability tractable is this: an agent is a workflow whose next-step function happens to be a language model. Everything else - tool execution, retries, approvals, timeouts, fan-out - is ordinary distributed systems work, and it deserves ordinary distributed systems machinery.

Most agent frameworks get this backwards. They treat the transcript as the source of truth: the loop appends messages, and "state" is whatever the context window currently holds. That works for a 90-second run. For a 5-hour run it means your durability story is "hope the process lives," your recovery story is "re-run everything," and your state is trapped inside a format (chat messages) that no external system can reason about.

The Microsoft Agent Framework's durable execution discussion frames the requirement well: agents need their execution state externalized so that orchestration - not the model, not the process - decides what has happened and what happens next. The model proposes steps. The workflow engine records them, executes them with retry semantics, and survives the host dying.

Once you accept that framing, the design follows. The state machine belongs outside the model context. The engine owns progress; the model owns judgment. Your agent loop becomes a workflow definition where one activity type happens to call an LLM, and the engine's existing guarantees - durable timers, retries with backoff, waiting days for a human signal - apply to agent steps for free.

Checkpoint the Task Graph, Not the Transcript

The most common durability mistake is snapshotting the wrong thing. Teams serialize the message history to Redis every few turns and call it checkpointing. Then a resume produces an agent that technically has its transcript back but has lost the structure of what it was doing: which subtasks are complete, which tool results are trusted, which action a human already approved.

Persist three things, in order of importance:

1. The task graph. The decomposition of the goal into steps, each with a status (pending, running, complete, failed, skipped). This is what lets a resumed run skip finished work instead of re-deriving the plan.
2. Tool results, keyed by step. The output of every completed side effect and retrieval. On resume, these are facts. The agent should never re-fetch or re-execute something the graph says is done.
3. Approvals and human inputs. If a person authorized a destructive action at hour two, that authorization must survive a restart at hour four. Losing an approval means either re-interrupting the human or, worse, an agent that proceeds without one.

The transcript is derived state. You can rebuild a working context from the task graph and tool results - a summary of completed steps, the current subtask, relevant results - and that rebuilt context is usually better than the original, because it is compact. This is the same insight behind context compaction for long-horizon sessions: the full history is an input to context construction, never the state itself. Zylos Research's analysis of durable execution for agent runtimes makes the same cut, separating replayable orchestration state from the model-facing view generated from it.

There is a cost argument hiding here too. If your checkpoint is the transcript, resume means re-feeding a bloated context. If your checkpoint is the task graph, resume means paying for one compact summarization call. Over a fleet of long runs, that difference is a real line item.

Idempotency Before Checkpoints

Here is the trap in checkpoint-and-resume thinking. Checkpoints protect the work you already did. They do nothing to protect the outside world from work being done twice. A crash can land after a side effect completes but before its checkpoint persists, and on resume the agent re-executes that step. The customer gets two emails. The card gets charged twice. The Jira ticket gets created again.

As HackerNoon put it, durable agent workflows need idempotency before they need checkpoints. The ordering matters because a non-idempotent agent with checkpointing fails worse than one without it: resumption converts a dead run into a duplicated side effect, and duplicated side effects are customer-visible in a way that dead runs are not.

The mechanics are standard, which is good news:

• Idempotency keys on every side effect. Derive the key from run ID plus step ID, so a retried step is deduplicated by the receiving system. Stripe's API has worked this way for a decade; most payment, email, and ticketing APIs accept a key.
• Check-then-act on systems without dedup support. Before creating the record, query for one tagged with your step ID. Slower, but it turns "at least once" into "effectively once."
• Classify tools by replay safety. Reads are safe to repeat. Writes need keys. Irreversible actions (sending, charging, deleting) need keys plus an approval gate. Tag this in the tool definition itself so the orchestrator can enforce it mechanically.

Agents make one part of this harder than classic workflows. A retried LLM call may produce a different plan than the original, so "retry the step" can mean "do a different thing." The fix is to treat the model's decision as a recorded value: once the model chose an action and that choice was persisted, retries re-execute the chosen action, never the choosing. Decision and execution are separate steps with separate durability. We covered the retry taxonomy in more depth in agentic workflow error recovery patterns.

What the Engines Actually Give You

You can build all of the above yourself with Postgres and discipline. Roughly nobody maintains that discipline past the third incident, which is why the durable execution engines are getting adopted under agent loops. Upstash's 2026 comparison of durable workflow engines covers the full field; the short version for agent builders:

The mechanism worth understanding before choosing is deterministic replay, because it is both the strongest guarantee and the sharpest constraint. Engines like Temporal do not snapshot memory. They record the result of every completed activity, and on recovery they re-run your workflow code from the top, substituting recorded results for completed steps. Nothing external executes twice, and recovery is exact. The catch is that workflow code must be deterministic - same inputs, same decisions - or replay diverges from history and the run fails.

An LLM call is the least deterministic thing you can put in a code path, so it must live inside an activity, never in workflow logic. The engine records the model's output the first time; replay reuses the recorded output. This lands exactly on the earlier point about idempotency: the architecture forces you to persist the decision and make the choosing itself a durable, non-repeated event. Restate's engineering posts on stateful agent sessions walk through the same pattern from the log-based side.

A useful heuristic for whether you need any of this - if your agent's longest expected run is under ten minutes and every side effect is reversible, a retry wrapper and a dead-letter queue are enough, and an engine is ceremony. Past ten minutes, past three external systems, or past the first approval gap measured in hours, the engine pays for itself the first time a deploy lands mid-run.

Retrofitting Durability Under an Existing Agent

Most teams are not starting fresh. They have a working agent loop in production and a growing list of runs that died partway, the kind of incident we dissected in when your agent failed in prod. Rewriting the loop inside a workflow engine on day one is usually the wrong move. The sequence that works:

1. Add idempotency keys to every write-class tool first. This is a tool-layer change, needs no engine, and immediately makes any future resume mechanism safe.
2. Externalize the task graph. Even a Postgres table of steps with statuses, updated as the loop progresses, turns "re-run everything" into "resume from step 14" with a manual script.
3. Move the loop into an engine only after the first two hold. At that point the migration is mechanical: steps become activities, the graph becomes workflow state, approvals become signals.

This is the shape of engagement where OpenNash spends most of its infrastructure time - not building a new agent, but putting a recovery spine under one that already earns its keep. The audit step maps which tools are replay-safe and which runs are actually being lost (teams routinely underestimate this because dead runs are silently restarted by hand). The build step does the three-stage retrofit above, usually landing on Temporal or Restate depending on the team's operational tolerance, and the handoff includes the runbook for the failure you will eventually hit anyway: a run that cannot resume and must be compensated. OpenNash deliberately does not push a house engine; the right answer depends on your stack, and the client owns whatever we deploy.

If you want the platform-versus-custom cut honestly: teams already deep in Azure should ride Durable Functions and the agent-framework work rather than adopt a second orchestrator. Teams whose longest run is minutes should wait - durability machinery ages badly when it is idle. Custom retrofits make sense when runs are hours long, side effects are irreversible, and losing a run has a dollar figure attached.

Where to Start This Week

Pull the last thirty days of agent runs and answer two questions: how many terminated without completing, and what did the longest successful run cost in tool calls and tokens? Multiply those together and you have the monthly price of non-durability. If the number is small, add idempotency keys anyway (they are cheap) and stop there. If the number makes you wince, the task graph externalization is a one-sprint change, and it is the prerequisite for everything else. If you would rather compress that timeline with someone who has done the retrofit before, book a durability audit with OpenNash and bring that thirty-day number to the call - it is the first thing worth looking at together.