Durable Execution for AI Agents: Checkpoints, Retries, Approvals

Durable execution means an agent's progress is written to storage as it runs, so a crash, a deploy or a three day wait for a human approval resumes the work instead of losing it. It only stays safe when every tool call that changes something is idempotent, retried under a clear policy, and gated by an approval that expires.

Agents now run for hours, call tools that send email, move money and open pull requests, and pause for people who are asleep. This guide explains how workflow engines and agent frameworks persist state, why retries become duplicate side effects without idempotency keys, how sagas undo what cannot be retried, how approval waits work, and which security problems appear once agent state lives in a database. It ends with a step by step design for an agent that drafts a pull request, waits for approval and retries safely.

Key takeaways

• Durable execution persists progress (an event history or step checkpoints) so a restarted process continues from the last completed step.
• Workflow engines replay a recorded history through deterministic code. Agent frameworks save a state snapshot per step, keyed by a thread ID.
• Every durable system retries, and retries repeat side effects. Treat tool calls as at least once and make each one idempotent with a key derived from the workflow and step.
• The IETF Idempotency-Key draft (draft 08, 22 October 2026) defines the key, its expiry, a payload fingerprint and the 400, 409 and 422 responses.
• Actions that cannot be retried or reversed need a compensating action (the saga pattern, 1987) or a human approval in front of them.
• A wait can be long. An approval should not be. Bind it to a hash of the exact action, give it an expiry, and re-check authority on resume.
• Checkpoints hold prompts, tool outputs and sometimes tokens. Encrypt them, keep secrets out, and never deserialize state with a format that can construct arbitrary objects.

What is durable execution, and why do long running agents need it?

Durable execution is a programming model in which the runtime records enough about a running process to rebuild it after a failure and carry on. The open source Temporal documentation says its workflows can keep running for years even when the underlying infrastructure fails, and that if the application crashes, Temporal rebuilds its pre-failure state and continues where it left off. The open source DBOS library checkpoints workflows and steps to Postgres, and after a crash or restart uses those checkpoints to recover each workflow from its last completed step.

A normal program keeps its progress in memory, and it disappears when the process dies. For a 200 millisecond web request that is fine. For an agent it is not, for four reasons.

Agents run long. A coding agent that plans, edits twenty files, runs tests and opens a pull request can take an hour. The longer a process runs, the more likely a deploy, an eviction or an out of memory kill will interrupt it.

Agents are expensive to restart. Restarting a forty step agent repeats forty model calls, and because model output varies, the second run may take a different path.

Agents act on the world. An agent that crashes halfway may already have sent an email or pushed a branch. Restarted from the beginning, it does those things again unless something remembers they happened.

Agents wait for people. Anthropic's guidance on building agents notes that agents "can then pause for human feedback at checkpoints or when encountering blockers." A pause that lasts until a reviewer wakes up cannot live in a process's memory.

Durable execution writes progress at defined points, records which side effects completed, and turns a wait into stored state. What it does not do is make a non-idempotent action safe to retry. That gap is where most of the engineering happens.

How do workflow engines implement durable execution?

The details below come from the Temporal documentation because it describes the mechanism precisely. Other engines in the category use similar ideas under different names.

Event history and replay

Each Temporal workflow execution emits commands and processes events, and the events are recorded in an Event History. The history is an append only log: the execution started, an activity was scheduled, it completed with this result, a timer fired, a signal arrived. The docs describe this history as what lets an execution recover from a crash and keep making progress.

Recovery does not reload a memory snapshot. Temporal runs the workflow code again from the beginning and replays the history step by step, using it to steer the code back to the exact state it was in before. When replayed code reaches an activity that already completed, the engine hands back the recorded result instead of running it again. Timers "are recorded as events and don't 'wait' again during replay." The workflow function runs many times over its life, but side effects inside activities run once per recorded completion.

The service limits both the number and the size of items in each execution's history, so agents that loop for days need to start a fresh history with carried over state, which the Temporal docs cover under Continue-As-New.

Deterministic workflow code

Replay only works if the workflow code makes the same decisions every time against the same history. The docs are direct: "Workflow code must be deterministic to support replay." Branching on the current time, a random number or a live API response can send the replay down a different path from the original.

A model call is about as non-deterministic as code gets, so it belongs outside the workflow function. The docs say to put API calls, LLM invocations, database queries and other external interactions in Activities. The model's output is recorded the first time, and on replay the workflow receives the recorded output rather than asking again, which keeps the agent's reasoning trace stable across crashes. Engines provide replay safe versions of time, randomness and IDs, and changing workflow code while executions are in flight needs a versioning discipline.

Activities, retries and timeouts

Activities are where side effects happen. Temporal's docs say activity code may use any library and may be non-deterministic, and add: "We recommend that it be idempotent, so retries can be processed without duplicate side effects."

Activities retry automatically. The documented default retry policy:

Worked example: the gaps between attempts are 1, 2, 4, 8, 16, 32 and 64 seconds. The next doubling would be 128, but the cap holds it at 100 seconds from then on. Seven retries take 127 seconds of waiting. Ignoring the time each attempt takes, one hour allows about 42 attempts: the first, 7 retries in 127 seconds, then about 34 more at 100 second spacing in the remaining 3,473 seconds.

That suits a flaky read. It is dangerous for an activity that sends an email and times out before reporting success, because each retry may send another. The controls are a schedule to close timeout that bounds total time across attempts, and non-retryable error types that stop retries at once. The docs also say retrying a whole workflow "is not recommended due to the deterministic nature of Workflow replay."

Long activities can heartbeat. The docs describe heartbeat details used for checkpointing: the last details are handed to the next attempt, so a long job resumes from its last batch. That is a checkpoint inside a step, with the same idempotency question as any partial work.

Step checkpoints in the database

The DBOS documentation describes a library inside your application with Postgres as its only dependency: "There's no separate orchestration server." It adds one database write per step to checkpoint the step's outcome, plus two writes per workflow, one for its inputs and one for its output.

Worked example: a ten step agent workflow writes 10 step checkpoints plus 2 workflow records, 12 writes per completed run. If the process crashes after step 6, recovery reads the 6 recorded outcomes, skips those steps, and runs 7 to 10. Step 7, in flight at the crash, runs again from its start. Whether that is safe depends on whether step 7 is idempotent.

Both designs share one boundary. Anything that completed and was recorded will not run again. Anything running when the process died may run again.

How do agent frameworks implement checkpoints?

Agent frameworks started as libraries for calling models and tools in a loop, then added persistence. The open source LangGraph library documents its design in detail.

Threads, checkpoints and super-steps

A LangGraph checkpointer "saves a snapshot of graph state at each super-step, organized into threads." You compile the graph with a checkpointer and pass a thread_id in the run configuration; runs with the same thread ID continue the same thread. A super-step is one tick of the graph in which every node scheduled for that step runs, possibly in parallel. LangGraph also stores each finished node's output as a pending write, so if another node in the same super-step fails, the successful writes do not have to be recomputed on resume.

The persistence docs warn that with MemorySaver and InMemorySaver, "all checkpoints are lost" when the process restarts, and point production use at PostgresSaver. They advise keeping thread IDs under 255 characters, using a UUID or hash.

Because every super-step is saved, you can invoke the graph with a prior checkpoint_id. Nodes before that checkpoint are skipped because their results are saved, and nodes after it run again. That is useful for debugging, and it is also a way to repeat side effects on purpose.

Durability modes

LangGraph lets you choose when checkpoints are written:

Under exit, a crash mid run loses every step since the start, so a restart repeats tool calls. Under sync, a completed step is on disk before the next begins. That narrows the window for repeats but does not close it, because a tool call can succeed and the process die before the write.

Interrupts restart the node

LangGraph pauses with interrupt(), which raises a special exception the runtime catches before saving state. You resume by invoking the graph on the same thread with Command(resume=...), and that value becomes the return value of the interrupt call.

The docs' "Rules of interrupts" add the catch: on resume the runtime restarts the whole node from its beginning, not from the line that called interrupt, so any code before the interrupt runs again. They state that "side effects called before interrupt must be idempotent," recommend placing side effects after the interrupt or in separate nodes, and warn against wrapping interrupt() in a broad try and except. It is the activity rule again from a different design. Something will run twice; choose what, and make it safe.

How a framework checkpointer differs from a workflow engine

A workflow engine records an event per side effect, and that record is what prevents an activity running again on replay. It also owns timers, retries with backoff, timeouts, external signals and a task queue that hands work to any worker. The price is a service to operate, deterministic workflow code and versioning.

A framework checkpointer records snapshots at step boundaries. By itself it does not retry with backoff across restarts, fire timers while no process runs, or route work across workers; something else must restart the process and invoke the thread. In exchange it is a library in your app, node code can be ordinary Python, and state is easy to inspect. Many teams combine the two. Either way, ask: when a crash happens mid tool call, what runs again, who restarts the work, and where does a pause live while nothing runs?

Approval in vendor agent SDKs

The OpenAI Agents SDK documentation describes a flow in which tools declare that they need approval, pending approvals appear on the run result as interruptions, and a serializable RunState lets you resume the paused run after decisions are made. A tool sets needs_approval to true or to a function that decides per call. The docs say needs_approval defaults to false for the local shell and apply patch tools, and that sticky decisions survive serializing and restoring the run state.

Anthropic's Claude Agent SDK documentation describes an ordered permission evaluation: hooks first, which "can deny the call outright," then deny and ask rules, the permission mode, allow rules, and finally a canUseTool callback that prompts at runtime when no earlier step has resolved the call. In plan mode, file edit and shell write tools go to the callback even when an allow rule matches.

Neither is a durable execution engine on its own. Both give you a serializable pause point and a place for the decision. Persisting the pause and making the actions after it idempotent is still your job.

Why must tool calls be idempotent?

Because every durable system retries, and retries repeat whatever the tool does.

At least once, at most once and exactly once

At most once means one attempt, never retried. No duplicates, but work can be silently lost.

At least once means retry until success is recorded. No silent loss, but if an attempt succeeded and the success was not recorded (a lost response, a killed worker, an early timeout), the next attempt repeats the action.

Exactly once is what everyone wants. Across a network boundary to a system you do not control, a runtime cannot provide it alone: it cannot tell "the mail server never got the request" from "it sent the email and the acknowledgement was lost." Exactly once effect comes from at least once execution plus an idempotent receiver that recognises the second request.

So the rule for agents: assume every tool call that changes something will be attempted more than once, and make the second attempt harmless.

Idempotency keys

Some operations are naturally idempotent: setting a field, upserting by natural key, adding a label already present. The IETF draft repeats RFC 9110's classification: OPTIONS, HEAD, GET, PUT and DELETE are idempotent, while POST and PATCH are not.

For the rest there is the idempotency key. The HTTPAPI working group draft "The Idempotency-Key HTTP Header Field" (draft-ietf-httpapi-idempotency-key-header-08, 22 October 2026) says the header "can be used to make non-idempotent HTTP methods such as POST or PATCH fault-tolerant." It is still an Internet Draft, not an RFC. It sets out:

• The key is a Structured Field item; a UUID or similar random identifier is recommended.
• The resource may expire keys and should publish its expiry policy.
• The resource may compute a fingerprint from the payload to detect a key reused with different content.
• Missing key: reply 400. Key reused with a different payload: reply 422. Retry while the original is still processing: reply 409 Conflict.

Its security considerations matter for agents. Resources that do not validate keys risk injection through the cache lookup, and low entropy keys let attackers guess other keys and read cached responses that belong to other clients. The draft recommends a published key format, validation of every key, and a composite cache key combining the client's key with attributes known only to the server.

For agents, where the key comes from matters most:

1. Derive it from the workflow, not the attempt. Workflow ID plus step name is the same on every retry and replay. A timestamp or fresh UUID per attempt protects nothing.
2. Do not let the model invent it. Generate it in code and record it with the step.
3. Bind it to the content. Include a payload hash, so a changed plan is a new operation rather than a silent replay of the old one.

Worked example: workflow wf-7f3a reaches step send_invoice_email with a payload whose SHA-256 begins 9c41. The key is a hash of wf-7f3a:send_invoice_email:9c41.... The first attempt reaches the mail service, which stores the key and sends, then the worker dies before recording success. The engine retries with the same key; the service returns the stored result and sends nothing. Now suppose a human comment made the agent revise the invoice, so the payload hash became e208. The key changes and the revised invoice goes out once. Had the agent reused the old key with the new payload, a server following the draft would reply 422, which is right: the mismatch is a bug and should stop the workflow.

If the tool takes no idempotency key, build the protection yourself: record the key with an "in progress" marker before the call and "done" after, and check the marker on retry. That narrows the window but cannot close it, since the call can succeed and the process die before "done." For that window you need a way to ask the remote system whether the action happened (find the pull request by its branch, the payment by its reference) or a compensating action.

What MCP tool annotations say about idempotency

The Model Context Protocol specification (2025-11-25) lets servers annotate tools. destructiveHint defaults to true. idempotentHint means that repeated calls with the same arguments "will have no additional effect on its environment," and defaults to false. Both apply only when readOnlyHint is false.

The same specification says "clients MUST consider tool annotations to be untrusted unless they come from trusted servers," and that for trust and safety there should always be a human in the loop able to deny tool invocations. Use idempotentHint to decide whether a runtime may retry a tool automatically only for servers you trust, and default to "not idempotent" when it is absent.

Sagas and compensating actions

Hector Garcia-Molina and Kenneth Salem named the pattern in "Sagas," ACM SIGMOD 1987. They proposed breaking long lived transactions into a sequence of smaller ones, each paired with a compensating transaction that undoes it "from a semantic point of view." Either all transactions complete, or completed ones are compensated in reverse order.

"Semantic" is the key word for agents. You cannot unsend an email; you can send a correction. You cannot un-merge a commit others built on; you can revert it. The paper also asks what happens if a compensating transaction itself fails. Compensations need retries and idempotency as much as forward actions do.

Email, money and merges

The actions hardest to compensate most need an approval in front of them, and the approval must sit before the side effect, not inside the same step, or retry rules will repeat the side effect after approval.

How do human approval steps work in a durable workflow?

An approval step is a durable wait: the workflow records that it needs a decision and stops consuming resources until the decision arrives or a deadline passes.

Signals, updates and timers

In Temporal, Signals can be sent "from any Temporal Client, the Temporal CLI," or another workflow, and Signal-With-Start starts a workflow and delivers a signal together. The workflow waits on a condition with a timeout. Signal inputs land in the event history, so replay sees the same decision at the same point. The wait is a durable timer: no worker sits blocked, and the engine resumes the workflow on any worker when the signal or the timeout arrives.

Interrupts and resume

In LangGraph the node calls interrupt() with a payload, the checkpointer saves state, and the run ends. Your application shows the payload to a person and invokes the graph on the same thread with Command(resume=decision). Nothing waits in memory, but nothing fires a timer either. If approvals should expire, your application must enforce it, by storing a deadline in state and checking it on resume, or by sweeping pending threads on a schedule.

How long can an agent wait safely?

Technically, as long as you like. The limit is how fast the world the approval describes goes stale:

• The input changes. The base branch moves; the invoice is corrected. An approved diff may conflict on Thursday, or apply cleanly and mean something different.
• The approver's authority changes. A decision made with authority on Monday can be replayed without it on Thursday.
• Credentials change. A token stored at the start either fails after the wait or succeeds with permissions policy has since withdrawn.
• The risk changes. A dependency the agent proposed may have had a vulnerability published meanwhile.

The safe pattern has four parts:

1. Bind the approval to the exact action. Put a hash of the diff and base commit, or the amount and payee, in the request, and have the decision carry it. On resume, recompute and refuse if it changed.
2. Give it an expiry. Set a deadline at creation and treat later decisions as void. Hours suit payments; a few days may suit a dependency update pull request. Write the number down and escalate or cancel when it passes.
3. Re-check on resume. Confirm the approver still holds the required role, fetch a fresh short lived credential, and re-validate key inputs.
4. Record the decision. Who approved what, when, against which hash, and what the agent then did.

What are the security issues with durable agent state?

OWASP's Top 10 for LLM Applications lists Excessive Agency (LLM06:2025): damaging actions taken in response to unexpected, ambiguous or manipulated model output, with root causes that include excessive permissions and excessive autonomy. Durable execution can amplify both, because the agent keeps acting long after anyone was watching.

Serialized state is an attack surface

A checkpoint is data written by one process and read back by another. If an attacker can influence what is written and the reader trusts it, the checkpoint store becomes a code path. Agent state is exposed because much of it is text from outside: web pages, tool outputs, file contents.

GitHub advisory GHSA-wwqv-p2pp-99h5 (CVE-2025-64439), published 5 November 2025 and rated high, shows how. It describes remote code execution in langgraph-checkpoint versions before 3.0.0. According to the advisory, the default JsonPlusSerializer used msgpack but fell back to a "json" mode when illegal Unicode surrogate values made msgpack fail, and that mode supported a constructor style format for rebuilding custom objects; an attacker who got a malicious payload persisted that way could run arbitrary code on deserialization. Version 3.0.0 stops deserializing custom objects saved in that mode, and langgraph-api 0.5 or later is not affected. See our post on the LangGraph checkpoint RCE for more.

The lesson is older than agents. The OWASP Deserialization Cheat Sheet says "Unpickling can execute arbitrary code; never unpickle untrusted data," and the Python docs warn "The pickle module is not secure." LangGraph's docs describe a pickle_fallback serializer option; turn it on and every checkpoint row must be trusted as much as your own code.

Secrets in checkpoints and history

The easy mistake is putting a token in agent state because a later node needs it. It is then copied into every checkpoint, retained for the life of the thread, and readable by anyone with access to the table.

Workflow histories record activity inputs and results and signal payloads too. Temporal's docs recommend a Payload Codec so data is unencrypted only in the client and worker processes on hosts you control, and publish what the codec covers. They also note that "Failure messages and call stacks are not encoded as codec-capable Payloads by default." An exception message containing a token lands in history as plaintext unless you enable that.

LangGraph documents an EncryptedSerializer that can encrypt all persisted state, reading its key from LANGGRAPH_AES_KEY. That protects backups and database readers, not code inside the worker that holds the key. The stronger rule: keep secrets out of state entirely. Store a reference and fetch the credential inside the step that uses it, with the shortest lifetime available.

Replay of stale permissions

A workflow that captured "this user may deploy" at the start and acts days later acts with permissions that may no longer hold. Replay tools make it worse: re-running later nodes from an old checkpoint repeats their tool calls against today's world on yesterday's reasoning. Check permissions at each consequential tool call with current identity and policy, and restrict and log who may replay production threads.

Deserialization of untrusted state

State crosses trust boundaries: an approval UI round trips serialized run state, a multi tenant service stores many users' threads in one table, a debugging tool edits state and resumes. Controls:

• Use data only formats and refuse object construction from stored data; the OWASP cheat sheet recommends allowlisting classes.
• Sign serialized state that leaves your server and verify it before resuming. State handed to a browser and back is user input.
• Scope thread IDs to tenants and check ownership on every resume.
• Keep checkpoint libraries patched; the fix above came in a major version.

Prompt injection that persists

Instructions planted in a page the agent read land in its state and are replayed into every later model call on the thread. A durable agent does not forget a successful injection when it restarts; it reloads it. Human approval of consequential actions is the backstop, because the approver sees the action the injection produced.

Comparison: four ways to run an agent

A thirty second read only research agent does not need a workflow engine. An agent that moves money across three systems with two approvals should not run in a loop in memory.

Step by step: an agent that drafts a pull request, waits for approval and retries safely

This dependency update agent reads a finding, prepares a fix in an isolated working copy, asks a human to approve the exact diff, then pushes a branch and opens a pull request. It must survive a crash at any point without opening two pull requests, acting on a stale approval, or writing a token into checkpoints.

The code is illustrative, to show the shape of the design. It is not production code; check each library's current documentation for exact signatures.

Step 1: separate pure steps from side effects

1. plan: read the finding and repository, decide the change. No writes outside the sandbox.
2. prepare_patch: apply the change in a temporary copy, run tests, compute the diff and its hash, set the approval deadline.
3. request_approval: pause.
4. verify_approval: check the decision.
5. push_branch and open_pull_request: the only external side effects, both idempotent.
6. record_outcome: write the audit record.

Steps 1 and 2 can rerun after a crash at the cost of tokens only.

Step 2: derive a stable idempotency key

# Illustrative example, not production code.
import hashlib

def step_key(workflow_id: str, step: str, payload_sha256: str) -> str:
    raw = f"{workflow_id}:{step}:{payload_sha256}".encode()
    return hashlib.sha256(raw).hexdigest()[:32]

# The workflow ID comes from the triggering event (finding ID plus repository),
# never from the clock or from model output.
workflow_id = "finding-4821:org-example/payments-api"

Step 3: store checkpoints encrypted, with no secrets in state

Compile with a persistent checkpointer and an encrypting serializer, as the LangGraph checkpointer docs describe.

# Illustrative example, not production code.
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.checkpoint.serde.encrypted import EncryptedSerializer

serde = EncryptedSerializer.from_pycryptodome_aes()   # reads LANGGRAPH_AES_KEY
checkpointer = PostgresSaver.from_conn_string(DB_URI, serde=serde)
checkpointer.setup()

graph = builder.compile(checkpointer=checkpointer)
config = {"configurable": {"thread_id": workflow_id}}

class State(TypedDict):
    repo: str                 # "org-example/payments-api"
    base_sha: str             # commit the patch was built on
    diff_sha256: str          # hash of the exact diff
    approval: dict | None     # decision, approver, decided_at, diff_sha256
    approval_expires_at: str
    pr_url: str | None
    github_installation: str  # an installation ID, never a token

Load the AES key and database URL from your secret store at start up, keep pickle_fallback off, and pin the checkpoint library at a version with the fix for GHSA-wwqv-p2pp-99h5 (3.0.0 or later).

Step 4: pause for approval with an expiry

# Illustrative example, not production code.
from langgraph.types import interrupt

def request_approval(state: State):
    # No side effects before interrupt(): this node re-runs on resume.
    decision = interrupt({
        "repo": state["repo"],
        "base_sha": state["base_sha"],
        "diff_sha256": state["diff_sha256"],
        "expires_at": state["approval_expires_at"],
    })
    return {"approval": decision}

The approval UI shows the diff, base commit and deadline, and returns a decision carrying the diff hash the approver saw. In a workflow engine the same step is a signal plus a timed wait, and the engine owns the timer:

# Illustrative example, not production code (Temporal Python SDK style).
@workflow.signal
def decide(self, decision: dict) -> None:
    self.decision = decision

# inside the workflow run method
try:
    await workflow.wait_condition(lambda: self.decision is not None,
                                  timeout=timedelta(hours=72))
except asyncio.TimeoutError:
    return await self.cancel_and_record("approval expired")

Step 5: verify the approval on resume

# Illustrative example, not production code.
from datetime import datetime, timezone

def verify_approval(state: State):
    d = state["approval"]
    if d is None or d.get("action") != "approve":
        raise Rejected("not approved")
    if datetime.now(timezone.utc) > datetime.fromisoformat(state["approval_expires_at"]):
        raise Rejected("approval expired")
    if d["diff_sha256"] != state["diff_sha256"]:
        raise Rejected("approved diff does not match prepared diff")
    if not approver_has_role(d["approver"], state["repo"], "maintainer"):
        raise Rejected("approver no longer authorised")
    return {}

Reading the clock is fine in a graph node. In a workflow engine, use its replay safe time function or put the check in an activity.

Step 6: make the side effects idempotent

Rebuild the commit from the same base, patch, author and dates, so a retry produces the same commit and re-pushing it changes nothing. If a retry ever produces a different commit, the push is rejected as non fast forward: stop and investigate rather than force push.

# Illustrative example, not production code.
# KEY was computed in code by step_key(workflow_id, "push", diff_sha256)
BRANCH="agent/fix-${KEY}"
git fetch origin main
git switch -C "$BRANCH" "$BASE_SHA"
git apply --index approved.patch
GIT_AUTHOR_DATE="$PATCH_TIME" GIT_COMMITTER_DATE="$PATCH_TIME" \
  git -c user.name="fix-agent" -c user.email="fix-agent@example.com" \
  commit -m "Update vulnerable dependency (workflow $WF)"
git push origin "HEAD:refs/heads/$BRANCH"

Before creating a pull request, look for one on the same head branch. GitHub's REST API documents a head filter on the list endpoint, and the create endpoint can return 422 on validation failure.

# Illustrative example, not production code.
existing=$(gh pr list --head "$BRANCH" --state all --json url --jq '.[0].url')
if [ -n "$existing" ]; then
  echo "$existing"          # a retry returns the first attempt's result
else
  gh pr create --draft --head "$BRANCH" --base main \
    --title "Update vulnerable dependency" \
    --body "Opened by workflow $WF. Idempotency key: $KEY. Approved diff: $DIFF_SHA"
fi

Check then create still races if two workers run the step at once, so allow only one active execution per workflow ID, or take a database lock on the key first.

Step 7: fetch credentials at the moment of use

Inside push_branch and open_pull_request, request a short lived installation token, use it, and let it expire. Never return it from the step or put it in state, and keep it out of exception messages, which may be persisted.

Step 8: set a retry policy that matches the action

# Illustrative example, not production code (Temporal Python SDK style).
open_pr_retry = RetryPolicy(
    initial_interval=timedelta(seconds=2),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=2),
    maximum_attempts=6,
    non_retryable_error_types=["ValidationFailed", "Forbidden", "Rejected"],
)

The waits between attempts are 2, 4, 8, 16 and 32 seconds, 62 seconds across five retries. Permission and validation errors stop at once. Rate limits and server errors retry, and because the step is idempotent, a retry after a lost response returns the existing pull request.

Step 9: record the outcome and test the crash paths

Write an audit record with workflow ID, approver, diff hash, branch, pull request URL and time. Then test the failures: run the workflow, kill -9 the worker after the push and before the pull request is recorded, start a new worker, and confirm exactly one pull request exists. Repeat with the kill after creation and before the step records success. Let an approval expire and confirm the workflow cancels.

Common mistakes

Non-deterministic code in replayed workflows. Reading the clock, generating IDs or calling the model directly in workflow code breaks replay. Use replay safe functions and put external calls in activities.

Raw API tokens in agent state. They are copied into every checkpoint. Store references and fetch credentials inside the step.

Retrying non-idempotent actions without limits. An unlimited policy on a step that sends email will eventually send twice. Add keys, bounded attempts and non-retryable errors.

Keys built from the attempt. A timestamp or per-call UUID differs on every retry and prevents nothing.

Side effects before an interrupt. Where resume re-runs the node, those writes happen again. Move them after the approval, into their own node.

Pending approvals with no expiry, or not bound to content. An open ended "yes" can be applied to a different diff, amount or recipient, long after anyone remembers the context.

In memory checkpointers in production. They lose every thread on restart.

Trusting annotations from untrusted servers, and leaving replay of production threads open to anyone.

Where Safeguard fits

Safeguard does not run workflows, store checkpoints or provide durable execution. It is a software supply chain security platform, so its fit here is indirect.

The agent runtime is software with dependencies: the framework, the checkpoint library, the workflow SDK, the database driver. Safeguard scans repositories, container images and SBOM files and matches components against vulnerability databases, and the platform shows which projects contain a given component and version. That answers "which services still run a checkpoint library older than 3.0.0" without opening each repository. Autonomous remediation can open fix pull requests, and it ships switched off.

Safeguard's own agent tools use the approval model this guide recommends. Since CLI version 11.10.0 the CLI asks before any command that changes something, and /permissions sets how much runs without asking. The desktop and phone apps can send work to a computer running the CLI and approve each command. MCP Guardrail, which generates guardrail proxy URLs for MCP servers to govern agent tool traffic, is generally available. Related reading: human approval gates for risky agent actions, what an AI agent kill switch must cover and multi-agent orchestration patterns.

FAQ

What is durable execution in simple terms?

Running code so its progress is saved as it goes. After a crash or restart, the runtime rebuilds where it was and continues instead of starting over.

Is durable execution the same as checkpointing?

Checkpointing is one way to implement it. Workflow engines usually replay an event history through deterministic code; frameworks and some libraries save state snapshots or step outcomes. Both skip work that already finished.

Do AI agents need a workflow engine?

Not always. Short read only agents are fine in memory, and a framework checkpointer covers most graph agents. A workflow engine earns its complexity when runs last days, touch several systems, need durable timers, or move money or code.

Can durable execution guarantee exactly once tool calls?

Not on systems it does not control. Runtimes give at least once execution plus a record of completed steps. Exactly once effects come from idempotent tool calls, usually via a key the receiver honours.

How do I make an LLM tool call idempotent?

Generate a key in code from the workflow ID, step name and payload hash. Send it if the tool accepts one, or record it with an in progress marker before the call, and check for an existing result on retry.

What happens to a LangGraph interrupt when the server restarts?

With a persistent checkpointer the pause is in the database. Invoking the graph on the same thread with Command(resume=...) re-runs the interrupted node from its start and returns your value. With an in memory checkpointer the pause is lost.

How long should an agent approval request stay valid?

There is no standard number. Set a deadline per action type, bind the approval to a hash of the exact action, and re-check inputs and authority on resume.

Does the MCP idempotentHint annotation mean a tool is safe to retry?

It is a hint, false by default. The MCP specification says clients must treat annotations as untrusted unless the server is trusted.

Durable agents are built from ordinary open source parts, such as workflow SDKs, checkpoint libraries, serializers and database drivers, and a flaw in one sits under every workflow you run. The checkpoint serializer advisory above was fixed in a major version, exactly the kind of upgrade that waits in a backlog, so the gaps below apply directly to your agent stack.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Gold Registry provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

These demos show Safeguard's own agent tools and approval flows on ordinary code; they do not run durable workflows. You can download Safeguard and follow the CLI docs to try them.

This is a real CLI session: install, open the interactive terminal and ask it to fix a lodash CVE. The assistant reads the finding and advisory, asks before any command that changes something, edits package.json, runs npm install and npm test with approval, and pushes nothing. The same flow applies when the vulnerable package is a checkpoint library in your agent service.

Click through it at your own pace, or open it full screen.

This real Android session sends work to a computer running the CLI and approves the command on the phone with Allow once, Allow for turn or Deny. It shows an approval reaching a person away from the terminal, the human side of a durable approval wait.

Click through it at your own pace, or open it full screen.

This demo connects Claude to the Safeguard MCP server with OAuth and chooses which tools Claude can use. That is the same least privilege choice you make about which tools a durable agent may call without asking.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• Temporal documentation: Workflows
• Temporal documentation: Activities
• Temporal documentation: Workflow Definition, determinism and constraints
• Temporal documentation: Events and Event History
• Temporal documentation: Retry Policies
• Temporal documentation: Sending Signals, Queries and Updates
• Temporal documentation: Codecs and Encryption
• DBOS documentation: Architecture
• LangGraph documentation: Persistence
• LangGraph documentation: Checkpointers
• LangGraph documentation: Interrupts
• IETF draft-ietf-httpapi-idempotency-key-header-08
• IETF Datatracker: Idempotency-Key header draft
• Garcia-Molina and Salem, Sagas, ACM SIGMOD 1987
• Model Context Protocol specification 2025-11-25: Tools
• Model Context Protocol schema 2025-11-25
• GitHub Advisory GHSA-wwqv-p2pp-99h5 (CVE-2025-64439)
• OWASP Deserialization Cheat Sheet
• OWASP Top 10 for LLM Applications 2025: LLM06 Excessive Agency
• Python documentation: pickle
• Anthropic: Building effective agents
• Claude Agent SDK documentation: Configure permissions
• OpenAI Agents SDK documentation: Human-in-the-loop
• GitHub REST API documentation: Pull requests

Last reviewed: 25 October 2026.