LLM API Timeouts, Retries and Jitter, Done Right

Three properties make model APIs different from a typical internal service.

Failures are often about capacity, not bugs. Claude returns 529 overloaded_error when the API is under high load across all users, and 429 rate_limit_error when your organization exceeds its limits. OpenAI returns 503 server_is_overloaded when a model lacks capacity, and 429 with a slow_down code when your request rate rises too fast. All of these are worth retrying, but only after the pressure has eased.

Calls are slow and expensive. A single request can run for minutes, and a retried request is billed again. A policy that retries too eagerly turns one slow failure into several slow failures.

Responses stream. Once tokens have reached the user, the request cannot simply be retried: the new response would start from the beginning and likely differ.

A good policy answers three questions for every failure: should this be retried at all, how long to wait first, and is there still time and budget to try again. The rest of this guide answers each one.

Start from the status code, then check the error body, because the status alone is ambiguous.

Retry: connection errors and timeouts, 408, 429 rate limits, 500, 502, 503, 504, and Claude's 529. Claude's docs say to retry 500 errors with exponential backoff. OpenAI's say to retry 500s after a brief wait, and to follow Retry-After on 429 and 503.

Never retry: 400 (bad request), 401 (authentication), 403 (permission), 404, and 413 (request too large). The same request will fail the same way; fix it instead.

The 429s that never succeed. Both providers also use 429 for limits that a retry cannot clear:

• On Claude, reaching your tier's monthly spend cap returns 429 rate_limit_error, the same type as a rate limit, but with no retry-after header and error.details.error_code set to enforced_spend_limit_reached. The docs state that retrying, including the SDK's automatic retries, fails until access resumes.
• On OpenAI, 429s with the codes credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded and organization_usage_limit_exceeded mean billing or quota limits. OpenAI's guide says retrying them won't restore access.

Needs a decision: a 409 conflict (Claude's docs say to resolve the conflict, then retry), and any error after a stream has started sending text, covered later.

The boundary is that error bodies change. Claude's docs say the type values may grow over time, and OpenAI moved video overload from 429 to 503. Treat unknown types as non-retryable, and log them so you notice new ones.

Before writing anything, know what you get for free. The anthropic (1.15) and openai (3.35) Python SDKs share the same retry design:

• Two retries by default (DEFAULT_MAX_RETRIES = 2), configurable with max_retries on the client or per request.
• Retried conditions: connection errors and timeouts, plus 408, 409, 429 and all 5xx responses. A non-standard x-should-retry response header overrides the decision when the server sends it.
• Backoff: 0.5 seconds doubled on each retry, capped at 8 seconds, multiplied by a random factor between 0.75 and 1.0.
• retry-after: both read retry-after-ms first, then retry-after as seconds or an HTTP date, and wait exactly that long when it is present. The OpenAI SDK does not retry at all when retry-after exceeds 120 seconds.
• Timeouts: 600 seconds by default, with 5 seconds to connect.

For many applications, the defaults are enough. They retry the right errors, respect the server's hints and stop after three attempts. Change them when you need something they do not do: a total deadline across retries, refusing to retry spend-cap 429s, different handling for streams, or more attempts for a background job.

The jitter is the weakest part for large fleets. A 0–25% reduction spreads retries over a small window, so many clients failing together still retry close together; the next section measures how much that matters.

Exponential backoff waits longer after each failure: base × 2^attempt, capped at a maximum. It reduces how often each client retries, but if many clients failed at the same moment, they all retry at the same moments too. Marc Brooker's AWS article shows the result: backoff without jitter still produces clusters of calls with gaps between them, so contention barely falls.

The fix is randomness. The article compares three schemes, using v = min(cap, base × 2^attempt):

• Full jitter: wait a uniform random time between 0 and v.
• Equal jitter: wait v/2 plus a random time between 0 and v/2.
• Decorrelated jitter: wait a random time between base and three times the previous wait, capped.

In its simulation, full and equal jitter did about the same amount of work and decorrelated did more. Equal jitter took much longer to finish, and the author calls equal jitter the loser and no-jitter backoff the clear loser.

To see the effect for a model API, this guide simulated 1,000 clients that fail at the same instant and retry four times, with base 1 second and cap 30, counting requests in the busiest 100 ms window. With no jitter, all 1,000 retries land in the same window. The SDK-style 0–25% jitter (with its 0.5-second base) still puts 791 there. Equal jitter brings it to 212, and full jitter to 158. Full jitter also finished sooner than no jitter: the mean time of the fourth retry was 7.6 seconds against 15.0.

This is the worst case, since real clients rarely fail at exactly the same instant. But it is the case that matters, because a provider outage or a deploy makes many clients fail together.

When the server tells you how long to wait, believe it. Claude's rate-limit docs describe retry-after as the number of seconds to wait before retrying and say that earlier retries will fail. OpenAI's guide says to wait at least as long as Retry-After specifies and to use exponential backoff with jitter only when the header is missing. A random backoff shorter than the server's hint just spends an attempt on a request that will be rejected.

Two cases break the simple rule:

• No header. Not every 429 carries retry-after. Claude's spend-cap 429 never does, so a missing header on a 429 is a reason to check the error code before backing off.
• A wait longer than you can afford. If the server says 300 seconds and a user is waiting, sleeping is the wrong answer. Fail fast with a clear error, move the work to a queue, or fail over. The OpenAI SDK has this rule built in: it refuses to retry when retry-after is over 120 seconds.

The policy below applies the second rule as a deadline: before any sleep, it checks whether the elapsed time plus the wait would exceed the caller's budget, and raises immediately if so. That turns "wait 500 seconds" into a fast, explicit failure the caller can handle.

Here is the complete policy for the Claude Python SDK. It turns off the SDK's own retries, classifies errors by status and type, honours retry-after, uses full jitter, enforces a deadline, and treats streams separately:

1import random, time2import anthropic3 4# Turn off the SDK's own retries so this policy is the only one.5client = anthropic.Anthropic(max_retries=0, timeout=anthropic.Timeout(120.0, connect=5.0))6 7RETRY_STATUS = {408, 429, 500, 502, 503, 504, 529}8RETRY_TYPES = {"overloaded_error", "api_error", "rate_limit_error", "timeout_error"}9NEVER_RETRY = {"enforced_spend_limit_reached"}10BASE, CAP, MAX_ATTEMPTS, DEADLINE = 1.0, 30.0, 5, 120.011 12def error_info(err):13    body = err.body if isinstance(err.body, dict) else {}14    error = body.get("error") or {}15    return error.get("type"), (error.get("details") or {}).get("error_code")16 17def retryable(err):18    if isinstance(err, anthropic.APIConnectionError):  # includes APITimeoutError19        return True20    if not isinstance(err, anthropic.APIStatusError):21        return False22    kind, code = error_info(err)23    if code in NEVER_RETRY:24        return False25    # Errors inside a stream arrive with the stream's HTTP 200, so check the type too.26    return err.status_code in RETRY_STATUS or kind in RETRY_TYPES27 28def server_wait(err):29    headers = err.response.headers if isinstance(err, anthropic.APIStatusError) else {}30    for name, scale in (("retry-after-ms", 0.001), ("retry-after", 1.0)):31        try:32            return float(headers[name]) * scale33        except (KeyError, TypeError, ValueError):34            continue35    return None36 37def backoff(attempt, err, started):38    wait = server_wait(err)39    if wait is None:40        wait = random.uniform(0, min(CAP, BASE * 2 ** attempt))  # full jitter41    if time.monotonic() - started + wait > DEADLINE:42        raise err  # waiting would blow the caller's time budget43    time.sleep(wait)44 45def remaining(started):46    return anthropic.Timeout(max(1.0, DEADLINE - (time.monotonic() - started)), connect=5.0)47 48def create(**params):49    started = time.monotonic()50    for attempt in range(MAX_ATTEMPTS):51        try:52            return client.messages.create(**params, timeout=remaining(started))53        except anthropic.APIError as err:54            if not retryable(err) or attempt == MAX_ATTEMPTS - 1:55                raise56            backoff(attempt, err, started)57 58def stream_text(**params):59    started = time.monotonic()60    for attempt in range(MAX_ATTEMPTS):61        sent_any = False62        try:63            with client.messages.stream(**params, timeout=remaining(started)) as stream:64                for text in stream.text_stream:65                    if time.monotonic() - started > DEADLINE:66                        raise TimeoutError("stream ran past the deadline")67                    sent_any = True68                    yield text69            return70        except anthropic.APIError as err:71            # Once text has reached the caller, a retry would repeat it: surface the error.72            if sent_any or not retryable(err) or attempt == MAX_ATTEMPTS - 1:73                raise74            backoff(attempt, err, started)

Call create(...) and stream_text(...) with the same arguments as client.messages.create. remaining() passes what is left of the deadline as the per-request timeout, so a single slow call cannot use the whole budget and still be retried afterwards.

The table shows each scenario the stub server was scripted to produce. One result is worth noticing. The error raised inside the stream had status 200. The SDK builds the exception from the stream's original HTTP response, which succeeded before the error event arrived. A policy that checks only status_code would treat a mid-stream overloaded_error as non-retryable, which is why retryable also checks the error type.

A single timeout value is not enough, because the SDKs' timeouts are not total durations. Both SDKs use an httpx-style Timeout(600, connect=5). The HTTP client's documentation defines the read timeout as the maximum wait for a chunk of data, not for the whole response. A non-streaming call sends nothing until it finishes, so there the read timeout behaves like a total limit. A stream sends chunks continuously, and the API also sends ping events, so a stream can run far past the read timeout without tripping it.

So set three limits:

• Connect: a few seconds. If the host is unreachable you want to know immediately.
• Idle (read): the longest acceptable gap between chunks. For a stream, a gap longer than this means the connection is stuck.
• Deadline: the total time for the call and all its retries, enforced in your own code. The policy above checks it before every sleep and on every streamed chunk.

Long non-streaming calls have one more problem: some networks drop idle connections, and the client never receives an answer. Claude's docs recommend streaming or the Message Batches API for long requests, especially those over 10 minutes. The SDK enforces this: it raises an error if a non-streaming request's max_tokens implies more than 10 minutes. It estimates 60 minutes per 128,000 tokens, which works out to about 21,333 tokens. It also sets TCP keep-alive on its connections.

Claude's streaming docs warn that errors can arrive inside the event stream after the API has already returned 200. For example, an overloaded_error arrives as an error event, where a non-streaming call would have received HTTP 529. Whether to retry depends on what the caller has already seen:

• Before any text was delivered: retry exactly as for a non-streaming call. The caller never knows.
• After text was delivered: do not retry silently. A new request starts the response again from the beginning, so the user would see the opening twice, and sampling means the second version can differ from the first. Raise the error and let the caller decide: show a "regenerate" button, or retry and replace what was shown.

stream_text implements this with one flag, sent_any, set when the first chunk is yielded. The stub test shows both paths: an error before the first chunk was retried and the caller saw "Hello" once; an error after "Hel" was raised, and the caller saw "Hel" and an exception rather than "HelHello".

Tool use adds a sharper version of the same problem. If the stream contained a tool call and your code already ran it, retrying the request may run it again. Make tools with side effects idempotent, or record which tool calls already ran. For handling cancellation and partial tool calls in streams, see Streaming Tool Calls and Cancellation.

Retries add load at the worst moment: when a service is failing, every client sends more requests. Three practices keep that bounded.

Retry in one layer only. If a gateway retries three times, your service retries three times and the SDK retries three times, one user request can reach the API 27 times. Pick one layer, usually the one closest to the API that knows the deadline, and set max_retries=0 everywhere else.

Cap retries across the fleet. A per-request limit does not stop a fleet of workers from multiplying traffic during an outage. A retry budget allows retries only while they stay below a fraction of total requests, and a circuit breaker stops calling a failing dependency for a while. See Circuit Breaker Pattern.

Know what a retry repeats. A model call is safe to retry in the sense that it changes nothing, but it is billed again and may give a different answer. A tool call with side effects is not safe to retry unless it is idempotent. In an agent, retry the model request and guard the tools; see Build an Agent Loop From Scratch.

When retries are exhausted, fail clearly: return an error that says the model was unavailable, keep the request ID, and offer a way to try again later.

Build a local stub of the Messages API and break your policy on purpose:

1. Write an HTTP server that answers POST /v1/messages from a script of responses, and point the SDK at it with ANTHROPIC_BASE_URL. 2. Script the nine scenarios in the table above and record how many calls each one makes. 3. Run them with the SDK's default retries, then with llm_retry.py, and compare the call counts for the spend-cap and retry-after: 500 cases. 4. Make the stub send an error event after the first text delta, and print err.status_code in your handler. 5. Simulate 1,000 clients retrying four times with no jitter, SDK-style jitter and full jitter, and count requests in the busiest 100 ms window. 6. Add a gateway retry around your own policy and count the API calls for an always-529 scenario.

Pass criterion: spend-cap and 400 errors make exactly one call, a long retry-after fails without sleeping, a mid-stream error after text never repeats output, and your simulation shows full jitter with a lower peak than no jitter.