Three retry layers turn one agent request into 27 calls

A tool adapter can obey its retry limit and still help overload the service it calls. Put another retry loop above it and an SDK retry loop below it, and their limits multiply.

For one logical agent operation, suppose each of those three layers permits three total attempts. If every downstream attempt fails in a way each layer considers retryable, the dependency receives 27 calls. The agent may report only three tool attempts.

I would require a retry owner for each dependency operation before enabling automatic recovery. A local limit such as “try three times” is incomplete until the team knows which other layers can replay the same work.

This is an old distributed-systems failure with an extra place to hide in agent orchestration. Amazon’s Builders’ Library article on timeouts, retries and jitter describes retries multiplying across service layers and recommends a single retry point for many low-cost operations. Google Cloud Storage’s retry guidance explicitly warns against adding application retries without accounting for retries inside the client library. Neither source establishes the defaults of your agent framework. You still need to inspect the installed stack.

Count attempts at the dependency boundary

Use “attempts” to mean total invocations, including the first. In this example, three attempts means one initial call and at most two retries. Configuration names are not consistent across libraries, so translate them into this convention before doing arithmetic.

The synthetic stack has an agent orchestration loop, a tool adapter and a dependency SDK. Each layer calls the next synchronously. When the lower layer exhausts its attempts, it raises a transient error that the next layer also retries.

The SDK makes three dependency calls before returning failure. The adapter invokes that SDK operation three times, producing nine calls. The agent repeats the adapter operation three times, producing 27 calls in total.

That count assumes persistent retryable failure, no deadline cutting the run short, and no shared budget stopping a later attempt. It is an exhaustion count for this particular call tree, not a forecast of normal traffic. Early success reduces the count. Fan-out to several dependencies, parallel attempts or separate recovery jobs would require a different model.

An application trace showing three tool invocations cannot establish the physical request count. Instrument the SDK transport boundary or a controlled test dependency. Record both the logical operation and each physical attempt, with a parent relationship that survives retries. Keep unique operation identifiers in protected traces rather than creating an unbounded metric label for each request.

Run the nested retry fixture

Download agent-retry-amplification.py, inspect it and run it with Python 3.9 or later:

python3 agent-retry-amplification.py
python3 agent-retry-amplification.py \
  --test
python3 -O agent-retry-amplification.py \
  --test

The fixture uses only the standard library. It makes no network calls, invokes no model and does not sleep. It executes nested functions that raise a synthetic transient exception, then counts actual calls to the innermost function.

Its default run compares these cases:

Setting an outer layer to one attempt preserves its initial invocation while removing its retries. The repaired failure case still fails; it stops spending attempts after the chosen owner has exhausted its allowance. That is the intended result when the dependency remains unavailable.

Ten unit tests cover nested exhaustion, single-owner behaviour, immediate success, recovery within and beyond the budget, permanent errors and invalid inputs. The permanent-error case makes one dependency call because the retry wrapper catches only the synthetic transient exception.

This is a counting fixture, not a production retry library. It deliberately omits backoff, cancellation, deadlines and distributed accounting so the multiplication remains visible. A passing test proves the fixture’s call counts. It does not prove your SDK has retries disabled or your service can tolerate the resulting load.

Put recovery where the operation is understood

The SDK is a useful retry owner when it understands a single remote operation’s error contract and can expose its attempt count. In that arrangement, the adapter forwards the final outcome and the orchestrator does not automatically restart the same operation after SDK exhaustion.

Sometimes the application must own recovery because the unit of work spans several calls. It might need to reconcile a partial result before deciding which remaining step to execute. Then explicitly disable or tightly limit lower-layer retries and include their remaining attempts in the application budget. Do not restart the whole workflow merely because its final step returned an error.

Write down the owner and the budget for each dependency path. Include the exact SDK version and retry configuration, what counts as a retryable failure, and whether the limit includes the first call. Also record the deadline boundary and the destination of an exhausted operation. A durable queue that redelivers tomorrow is another retry layer even if today’s in-memory loop stopped correctly.

For an agent, returning a tool error to the model does not enforce exhaustion. The model may choose the same tool again, perhaps with slightly different wording. If the application requires a hard cap, enforce it outside the model against a stable logical-operation record. Decide which argument changes constitute a genuinely new operation; otherwise a fresh tool-call identifier can reset the budget without changing the work.

The earlier retry evaluation article concerns how eventual success can hide first-attempt failure. Here the acceptance question is different: how many physical requests can one logical operation generate when the dependency stays unavailable?

Jitter changes timing, not retry ownership

Backoff spaces attempts apart. Jitter spreads them across time so clients are less likely to retry together. Those mechanisms can reduce synchronised pressure, but adding them to all three loops does not, by itself, change this fixture’s maximum of 27 calls. Deadlines or admission limits may stop the sequence earlier; the retry allowances still exist.

Use the selected client’s supported retry policy rather than adding another generic decorator around it. Check how that client handles server retry instructions, maximum delay and total elapsed time. A long sleep can consume the caller’s remaining deadline before a useful attempt begins. Conversely, a caller timeout does not necessarily stop lower-layer work already in progress.

A per-operation cap also cannot protect a dependency from unlimited concurrent operations. During an outage, each new operation can spend its own allowance. Bound admission and concurrency separately, and consider a shared retry budget at the dependency level. If that budget spans processes, an ordinary in-memory counter in each worker does not enforce the global limit.

Read operations need this protection too. Idempotency can make repetition safe for resource state while repeated reads still consume capacity. For writes, first establish whether repetition is safe at all. An unknown outcome requires reconciliation or a supported idempotency contract; changing retry ownership cannot make a non-idempotent write safe.

Test the configured stack under persistent failure

Run this acceptance test against a disposable dependency, with the production adapter and SDK versions. Inject a retryable failure at the transport boundary and keep it active until the operation terminates. Compare the observed physical calls with the declared limit. Do not infer the result from the number of orchestration spans.

Then test a failure that the provider defines as permanent and verify it does not enter the transient retry path. Test recovery on the final permitted attempt. Exercise caller cancellation and deadline exhaustion separately, observing whether downstream attempts continue after the caller has stopped waiting. The timeout accounting article explains why caller-visible termination and backend completion need separate records.

For queued work, include redelivery and process restart in the test. Retain a receipt containing the logical operation, configured owner, observed physical attempts and final outcome. A request-count mismatch is a release defect even when the last attempt succeeds.

For the broader operating model, my book Claude Code: Building Production Agents That Actually Scale on Amazon Kindle covers tool governance, cost controls and review around agent workflows.

Start with one dependency path in your own stack. Set its expected exhaustion count before running the failure test. If the transport observer records more calls, trace the additional retry owner before increasing the allowance.