An agent fails a task, tries again and eventually produces a correct result. The report marks the task green.
That can be an honest measure of search capability. It becomes misleading when someone reads the same green cell as evidence that a customer can rely on the first attempt. The unsuccessful attempts still happened. They may have consumed review time or changed external state before the successful attempt began.
My position: a retry allowance is part of the product contract. If the evaluation gets three attempts but the customer expects one safe operation, the score describes a different service.
The distinction is measurable. This article includes a small offline Python fixture that reports first-trial, any-trial and all-trial success from the same records. Its results are synthetic teaching data, not measurements of Claude Code or any other model.
Two scores that answer different questions
Anthropic’s guide to agent evaluations distinguishes pass@k, the probability of at least one success in k attempts, from pass^k, the probability that all k trials succeed. The original tau-bench paper introduces the latter to examine consistency across repeated agent interactions.
Those questions are both useful. A developer generating candidate patches in disposable environments may care whether one candidate solves the problem. A support workflow that changes a customer’s account needs evidence about consistent behaviour, including the attempts that do not end well.
There is another condition on the candidate-patch example: the product must identify the correct candidate. An offline grader that sees the reference answer can choose a winner more accurately than the deployed selector. Finding one correct answer somewhere in the batch does not prove that the system will return it.
Report what happened at selection time. Did the selector choose a passing patch? How much did generating and testing the rejected candidates cost? Did any candidate bypass the permitted files or leave a changed shared environment? A best-of-several score alone cannot answer those questions.
Keep the task rows visible
Consider this deliberately small fixture. Each row represents the same task run three times from a reset starting state. A pass means the declared task contract passed, rather than merely that the final message sounded correct.
| Task | Trial 1 | Trial 2 | Trial 3 |
|---|---|---|---|
| A | Pass | Pass | Pass |
| B | Fail | Pass | Fail |
| C | Fail | Fail | Pass |
| D | Pass | Timeout | Fail |
All four tasks pass at least once: 4/4. Only A passes every scheduled trial: 1/4. The first-trial result is 2/4, and there are 6 passing trials out of 12 scheduled trials.
None of those counts contradicts another. They describe different properties of the same observations. Reporting only the first count erases the instability that the other counts expose.
These are observed task fractions, not the combinatorial estimators used by some benchmark implementations. The fixture gives every task exactly three scheduled trials and counts the observed groups directly. It neither estimates a production success probability nor supplies confidence intervals. Four invented tasks would be a poor basis for either claim.
The timeout remains visible. It is not a verified success, but this fixture does not pretend to know whether an external effect occurred before the timeout. For scoring, it prevents an all-trials-pass result. For incident handling, it requires investigation. Those are different decisions.
Run the counter before running more models
Download agent-repeated-trials.py. It uses Python’s standard library, makes no network calls and incurs no model charges.
python3 agent-repeated-trials.py
python3 -O agent-repeated-trials.py
Each command prints the synthetic counts and runs twelve distinct unit tests. The tests cover missing and duplicate trials, unexpected task identifiers, invalid outcomes, malformed input and explicit unknown results. Reversing the input records leaves the counts unchanged because trial identity comes from the task identifier and attempt number, not file order.
The relevant output is:
{
"tasks": 4,
"trials_per_task": 3,
"scheduled_trials": 12,
"first_trial_pass_tasks": 2,
"any_pass_tasks": 4,
"all_pass_tasks": 1,
"passed_trials": 6
}
The complete output also breaks down pass, fail, timeout, error and unknown outcomes. No-argument mode exits according to its unit tests. Both normal and optimised Python passed before publication.
For your own records, supply a JSON manifest with this shape:
{
"tasks": ["refund-check"],
"k": 2,
"trials": [
{"task": "refund-check", "attempt": 1, "outcome": "fail"},
{"task": "refund-check", "attempt": 2, "outcome": "pass"}
]
}
python3 agent-repeated-trials.py manifest.json
A valid manifest exits with status 0 even if every trial failed. That exit means the records satisfy the input contract, not that an agent is safe to release. Invalid input exits with status 1. Use the printed counts in a separately defined release policy.
The manifest declares the task inventory and repeat count before supplying results. Every expected pair must appear exactly once. A crashed worker cannot quietly reduce the denominator by failing to write a row. The collector must preserve its outcome explicitly, or the manifest fails validation.
This does not prove that the declared inventory is complete. An exporter could omit a task from both the inventory and the results. Bind the manifest to an approved task list outside the exporter, and preserve configuration and evidence references in your actual harness. The downloadable counter deliberately handles only a small reporting contract.
Repeated trials and product retries are different experiments
For repeated-trial reliability, reset the relevant environment between trials. Keep the task, model configuration and grader fixed. Do not let the second trial inherit the first trial’s answer or repaired database unless that inheritance is what the experiment is meant to test.
Anthropic’s evaluation guide also recommends isolated trial environments and warns that shared state can inflate performance or create correlated failures. The earlier article on an eval passing because a previous case cleaned up for it examines that contamination problem.
A product retry policy is another unit of evaluation. It may feed a failed test back to the model, retain a partial patch or ask the user for clarification. Evaluate that complete policy as one customer-visible workflow, with its own maximum attempts and deadline. Preserve the individual attempts inside the trace.
Do not call the attempts independent when later attempts receive earlier feedback. Even reset trials can share failure causes, such as a broken tool service or the same misunderstood requirement. Simple probability calculations based on independent, identical trials can therefore be inappropriate. Observed counts remain useful, but the uncertainty analysis must match the experiment.
For comparing two configurations, use the same declared task set and repeat policy. Keep per-task outcomes so a gain on easy tasks cannot conceal a regression on a consequential workflow. The slice-level release-gate example addresses that separate problem.
A successful retry cannot undo an unsafe attempt
Suppose a synthetic refund task permits one refund. The first attempt creates the refund but loses its response. A retry creates a second refund and reports success. A grader that checks only whether a refund exists can mark the workflow green while missing the duplicate effect.
That example is a design scenario, not a reported incident. Its lesson is that success criteria must include prohibited effects. Inspect the final ledger and relevant action records, not only the last reply.
Before retrying an uncertain write, reconcile the operation’s status. Use the target system’s documented idempotency mechanism where available, with a stable identity for the intended operation and protection against reusing that identity for different parameters. A fresh identifier on every retry defeats deduplication. A local counter cannot supply guarantees that the remote service lacks.
Likewise, a later correct answer does not erase an earlier disclosure. Safety failures need their own record and release policy. Whether a particular failure blocks a release is a risk decision made in advance, rather than something a higher best-of-several score should settle afterwards.
Put the retry policy next to the result
A useful release report names the task inventory and the model, harness and grader versions. It states the scheduled repeat count, whether environments were reset, and what feedback later attempts received. Then it presents the first-trial and repeated-trial counts alongside unresolved outcomes.
For workflows that deliberately retry, add the whole-workflow success rate, the selector’s observed performance where selection applies, and total latency and cost across all attempts. Keep permission violations and uncertain external effects visible even when the final result passes. Measure those quantities in the harness; this counter does not collect them.
My book AI Agent Evals develops the wider workflow, including task contracts, harnesses and comparisons under uncertainty. Get AI Agent Evals on Leanpub.
When the next report says a task passed, ask which attempt passed, what happened before it, and whether the deployed workflow can recognise the same success. Keep the failed attempts in the evidence.