Written by
Meryem Cebeci
Date added
Suppose an agent step calls a ticketing API to create a support ticket. The call times out after 30 seconds, the retry layer sends it again, and the second call returns a ticket ID. Did the customer end up with one ticket or two?
The timeout cannot tell you. It says no answer arrived in time, not that the work didn't happen. Worse, retries stack: an SDK, a framework and your own loop can each retry the same call without knowing about the others.
Below: a count of the requests and wall-clock time one step can use under default settings, a map of which timer owns which wait, a decision table for ambiguous timeouts, and a pre-release checklist.
A timeout is not a failure
AWS puts it plainly in its Builders' Library article on timeouts, retries and backoff with jitter: "A timeout or failure doesn't necessarily mean that side effects haven't happened." LangGraph's fault-tolerance docs give the agent version: if the provider commits and the acknowledgement is lost, "the graph has no local record even though the side effect may have happened."
For writes (an email, a ticket, a charge) this matters. A CrewAI forum post reports a "send the summary email" step firing four times from one request. That is one team's account, not a maintainer statement. Controlled evidence exists too. The preprint "Where Does Exactly-Once Live?" built a simulated sandbox (LIMBO) with six services and twelve fault modes, and ran 25,930 agent episodes across nine models. Its abstract reports that when a write's acknowledgement was lost but an immediate read-back could reveal the outcome, frontier models told to act exactly once almost never duplicated the write (0.5%). When the request was still in flight, or the transport delivered it twice, they duplicated in 56% and 74% of episodes. Agents also reported success in 90% of the episodes in which they had duplicated an effect. These are the authors' simulated results, not production data.
Two lessons follow. The agent's own summary is not a reliable record of what happened, and whether a retry is dangerous depends on what the failed attempt could have already done: a question about the target operation, not the error.
Count the attempts before you tune anything
Before choosing timeout values, check how many attempts one logical step can make. AWS gives the general version: five layers with three retries each means "the load on the database will increase 243x" (3^5), and its advice is to "retry at a single point in the stack." For agents the layers are concrete: two common SDK clients, a LangGraph node with a retry policy, and an application loop whose count we assume.
Layer | Attempts per call | What it retries |
|---|---|---|
OpenAI and Anthropic Python SDKs | 3 (1 + | HTTP 408, 409, 429, any 5xx, timeouts and connection errors; an |
LangGraph node with | 3 ( | Default |
Application loop around the step | 3 (assumed for this example) | Whatever you wrote |
Sources: openai-python, anthropic-sdk-python, LangGraph docs, LangGraph source
SDK values are from each repository's main branch as of November 2026; check them against your release. LangGraph retries are opt-in per node via retry_policy=.
The mismatch between layers is the part no single doc shows. Reading the source, both SDK exception hierarchies derive from plain Exception, not from httpx or the excluded built-in classes, so LangGraph's default retry_on treats them as retryable whatever their status. That is our reading of the source, not something we ran. If it holds, a node with a default policy retries the SDK's final 429 or timeout, and also a 400, 401 or 404 that the SDK itself refused to retry.
What that means in requests
SDK only: at most 3 HTTP attempts per call.
SDK plus a default node policy: the node retries the SDK's final error, so a retryable failure costs 3 x 3 = 9 requests. A 401 costs 3: the SDK sends it once and declines to retry, then the node runs it three times.
One layer owns retries: set the SDK client's
max_retriesto 0, or give the node aretry_onthat excludes SDK errors. The count drops to 3, and the layer that knows the operation's semantics decides.Three layers: the loop around the step adds a third factor, 3 x 3 x 3 = 27 requests for one logical step before any backoff, assuming every layer retries the same error class.
Based on: openai-python, anthropic-sdk-python, LangGraph docs
Time stacks the same way. The OpenAI SDK's default timeout is 600 seconds (5 for connect), and it retries timeouts. If the 600 seconds acts as an httpx read timeout, which httpx defines as "the maximum duration to wait for a chunk of data to be received", a response that never starts can wait 600 seconds per attempt. Three attempts is 3 x 600 = 1,800 seconds, about 30 minutes, plus under two seconds of backoff (0.5 s, then 1.0 s). Wrap that call in a three-attempt node policy with no node timeout and the worst case is about 90 minutes. This assumes the stall happens before the SDK hands the response back, and that the SDK's httpx-based package keeps httpx's semantics.
Based on: openai-python, HTTPX docs
If one layer can stall for half an hour, which timer actually ends the wait?
Which timer owns which wait
"Timeout" names four different things: connect, read (or per-chunk), a cap on one attempt, and a cap on the whole operation including retries. The first two bound a single wait, so a slow but steady stream can reset them indefinitely. Only the last two bound elapsed time.
Layer | Timer | What it bounds | Default |
|---|---|---|---|
OpenAI and Anthropic Python SDKs |
| per wait: connect, read, write, pool | 600 s; connect 5 s |
LiteLLM proxy |
| proxy-wide default; each wait for the next chunk on native | not stated in the docs read |
LangGraph node |
| one attempt; | not stated; needs |
Temporal activity | start-to-close | one activity task execution | none of its own; uses schedule-to-close if only that is set |
Temporal activity | schedule-to-close | the whole activity, including retries | infinite; every activity needs this or start-to-close |
AWS Application Load Balancer | idle timeout | silence between bytes on a connection | 60 s |
Sources: openai-python, anthropic-sdk-python, LiteLLM docs, LangGraph docs, Temporal timeouts, AWS ALB docs
Idle-style limits catch silent hangs and total caps bound the run, so you want both: a total cap alone cuts a reasoning stream that is still making progress. The MCP specification takes the same position: a sender may reset the timeout clock on progress notifications but "SHOULD always enforce a maximum timeout, regardless of progress notifications."
LangGraph expresses the split directly with TimeoutPolicy. Node timeouts apply only to async nodes, and sync nodes with a timeout are rejected at compile time.
A LangGraph NodeTimeoutError is retryable by default once a retry policy exists, so a node timeout plus a default policy can turn one hang into several.
Proxies have timers too
Not every cut comes from your client. An AWS Application Load Balancer's idle timeout defaults to 60 seconds, so a reasoning stream that is silent for longer can be cut by the proxy. Check the idle timer of every proxy or gateway between your agent and the provider.
Once you know which timer fired, the next question is whether the layer that saw the failure should retry. The SDKs make the first decision for you.
What the SDKs retry, and when they stop
Both Python SDKs retry on 408, 409, 429 and any status of 500 or above, plus timeouts and connection errors. Other statuses (400, 401, 404) are not retried. An x-should-retry header from the server overrides the decision. Both use backoff starting at 0.5 s, doubling, capped at 8 s, reduced by a random 0 to 25% of jitter (openai-python, anthropic-sdk-python). They differ on Retry-After: the OpenAI SDK does not retry if it exceeds 120 s, while the Anthropic SDK waits for the server's value with no 120 s cap. So a 429 with a long Retry-After ends quickly in one and sleeps in the other; handle long waits in your application if you use both. Jitter, as AWS notes, "adds some amount of randomness to the backoff to spread the retries around in time", so clients that failed together don't retry together.
Retry, verify or stop: what to do for each outcome
Counting attempts shows how much retrying happens by default. Deciding which retries should happen takes a different question: what could the failed attempt have already done? The table answers it by outcome. The "Action" column is our reasoning; the evidence column shows what supports it.
Outcome you observe | What it most likely means | Action | Evidence |
|---|---|---|---|
400, 401, 404 or another non-retry status | The server rejected the request | Do not retry. Fix the request or credentials. Make sure outer layers don't retry it either | SDK retry rules; LangGraph default |
408, 409, 429 or 5xx on a side effect | The server may or may not have acted | Retry only if the target honors a stable idempotency key or the operation is idempotent; otherwise verify first | AWS idempotency guidance; LangGraph docs |
Timeout or lost response on a side effect, target honors a stable key | Unknown outcome | Retry with the same key. This is the preferred path | LangGraph docs; simulated study (below) |
Timeout or lost response on a side effect, no key | Unknown outcome | Verify with a read-back within a bounded wait. If there is no read path, or the wait is unbounded, escalate or reconcile. Do not blind-retry | Simulated study; LangGraph docs |
Timeout on a read-only operation | Safe to repeat | Retry with backoff and jitter, within a budget | AWS: read-only APIs are typically idempotent |
Sources: AWS Builders' Library, LangGraph docs, "Where Does Exactly-Once Live?", openai-python, anthropic-sdk-python
Why keys come before waiting: in the same study, the authors report that offering an idempotency key on every write lowered the duplicate rate from 28% to 4%, because agents use keys when they exist. They also prove that no verification-only policy is exactly-once under late commits without a bound on in-flight time, and report that with heavy-tailed delays even an hour of waiting per episode fell short of keys. Our reading: waiting is a poor substitute for a contract with the target. A read-back helps when the outcome is visible quickly, as in the 0.5% case, and fails when a write commits after you have looked.
Read-back has another catch. A commenter on the CrewAI thread reports that a read can fail right after a write: "A read that finds nothing can simply be behind." It is one commenter's small test, but it is a reason to require a fresh read path before treating "not found" as "didn't happen".
The table doesn't apply when the target offers neither a key nor a read path. There the choice is between escalating and accepting a possible duplicate, a business decision, not a retry setting. A second preprint reports that a wrapper combining postcondition verification, verify-before-retry logic and idempotency keys reduced duplicate actions in a simulated environment (arXiv 2608.02645).
Idempotency keys that survive a retry
An operation is idempotent if repeating it has the same effect as doing it once. For a non-idempotent API, the usual route is an idempotency key: the server returns the saved result for a repeated key instead of acting again. In Stripe's documented behavior, retries with the same key return the saved response, a reused key with different parameters errors, and keys may be removed after at least 24 hours (API v1).
Two requirements matter for agents. The server must honor the key, so for your own tools you have to build that behavior. And the key must be reproducible from durable state: LangGraph's guidance is that "a retry or a resume in a fresh process reconstructs the same key". A random key generated inside the retried code fails this test.
One practitioner argues that keys derived from request content can differ between attempts, and prefers keys derived from the workflow position plus a durable "action ledger" the tool checks before acting (tianpan.co). That is one author's design view, but a useful test for your scheme: would two retries of the same step produce the same key?
Put retries in the right unit
The unit a framework retries decides whether a side effect can run twice. LangGraph retries whole nodes, so every side effect inside a retried node is repeated. Its docs advise: "For side effects that cannot be made idempotent at the API level, move them out of retryable nodes. For example, run them from an error handler after retries are exhausted, or from a separate worker that deduplicates by task ID."
Two settings follow from the attempt count. Give a node's RetryPolicy an explicit retry_on that retries only transient errors, rather than the default, which (from reading the source) also retries SDK-raised 4xx errors. And pick one layer for retries: if the node retries, set the SDK client's max_retries explicitly; if the SDK retries, keep the node's policy off for side-effecting calls.
Temporal's unit is the activity, where the side effect lives. Its activities carry a retry policy by default with unlimited attempts (Temporal retry policies), so set a maximum attempt count, list non-retryable error types, and give each activity a schedule-to-close or start-to-close timeout (Temporal requires one).
Set the budget for the whole run
Per-call limits do not bound a run. Decide a total budget and make each layer fit inside it, as Temporal's schedule-to-close timeout and the MCP maximum-timeout rule do. AWS adds a reason to be strict: retries are "selfish", spending more of the server's time for a better chance of your own success, so limit them with a retry budget, backoff and capped attempts.
See every attempt in one trace
Once the limits are set, you still need to see what happened in a run. Rius shows timeouts and retries in one trace; the fix stays in your application.
Instrument your model calls with the Rius SDK. With the OpenAI Python integration, a call that OpenAI rejects, such as a 404 for an unknown model, a 429 or a timeout, is recorded as an ERROR span with error.type naming the error class (Rius OpenAI integration). Trace each tool as a TOOL span: the Tool Loop rule counts TOOL spans only, and a bare observe is a CHAIN span.
Then open a run whose root span finished OK. A run is badged recovered when a child span errored but the root finished without an error of its own, and back-to-back calls of the same tool or step collapse into one row such as fetch_article · 3 attempts · recovered (Rius docs). That separates a retry that worked from a clean success.

A retry that recovered in Rius: the run succeeded and the flaky step stays visible (illustrative data).
For recurring problems, the Consistent Error Pattern alert groups errored spans by service, step and normalized error message, and by default fires at 3 or more errored spans across 2 or more distinct traces within 5 minutes. The docs name this case: "the same 429 or timeout recurring across runs while everything else stays green" (Rius pre-defined alerts). A single trace retrying the same step never fires it on its own.
For the hung-run case from the 30-minute calculation, enable partial spans (partial_spans, off by default; see the Rius reliability docs). A trace with a span still open then reads as running… (filter incomplete), and one stuck there long after the work should have ended "points to a crash, a hang, or a killed process" (Rius docs). You see the open span while the run waits, not after it times out.
Two more alerts concern repeated work. Swallowed Error fires when the root span finishes OK while at least one non-tool child step errored (errored tool calls don't count). Tool Loop fires when the same tool is called with identical parameters at least 3 times in a row inside one trace.
Rius shows the error spans, recovered runs and retry chains around a duplicate action; the key, read-back and limit decisions above still belong to your code. Given that agents in the simulated study reported success in 90% of the episodes where they had duplicated an effect, the trace is a better record than the agent's own summary.
Pre-release checklist
Does every remote call have a timeout, and do you know which timer fires first?
Have you counted attempts across SDK, framework and application, and does exactly one layer own retries?
Is there a total cap on the step and the run, not only per-wait timeouts?
Do streamed calls have both an idle limit and a total limit?
Have you checked the idle timer of every proxy on the path?
Do retry conditions list transient errors only, so 400, 401 and 404 stop at every layer?
Is every side-effecting tool idempotent or called with a key reproducible from durable state?
With no key, do you verify with a fresh read within a bounded wait before retrying?
Are side effects kept out of units a framework retries automatically?
Can you see each attempt, its error type and the run's verdict in a trace?
The question to ask before every retry is: what could this attempt have already done?
To see attempts, error types and recovered runs for your own agents in one trace, the Rius quickstart walks through connecting your first agent.
Sources and further reading
AWS Builders' Library: Timeouts, retries, and backoff with jitter
Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures (arXiv 2608.02645)
tianpan.co: Exactly-Once Was Hard Before Your Agent Could Retry Itself
Rius pre-defined alerts, recovered runs, SDK reliability, OpenAI integration
Share the article
You might also like

Agent Observability
How to detect silent AI agent failures in production
Fifteen signals and a five-stage framework for detecting silent AI agent failures while runs are still running
Written by
Armend Avdijaj
read article
read article

Agent Observability
How to Debug an AI Agent: A Step-by-Step Guide
How to debug an AI agent in production: find the run, trace it to the first divergence, classify the failure mode, and ship a fix. A step-by-step guide." slug: "blog/what-is-agent-observability
Written by
Pablo Pardo Garcia
read article
read article

Agent Infrastructure
AI agent failure modes: the seven ways agents fail in production
The seven ways AI agents fail in production: tool misuse, context loss, goal drift, retry loops, frozen agents, cascading errors & silent degradation.
Written by
Pablo Pardo Garcia
read article
read article





