Posts

Zero observed failures with 95% one-sided upper bounds: 30 trials gives 9.50%, 100 gives 2.95%, and 300 gives 0.99%. These bounds require independent trials with a fixed failure probability.

Your agent passed every test. How much failure did you rule out?

A clean evaluation needs a sample-size argument. Calculate what zero failures supports, and separate statistical uncertainty from missing or correlated evidence.

Synthetic example: 100 runs contain five failures. One retained subset has zero failures in ten traces; a failure-enriched subset has five in fifteen. Neither raw subset rate describes all runs.

Your agent failure rate depends on which traces you kept

A failure-enriched trace store is useful for investigation. Its raw failure percentage is a poor release metric. Keep an independent outcome denominator.

Partition 10,000 input tokens into ordinary input, cache reads and cache writes. Count 1,000 output tokens once, including their 700 reasoning tokens.

Your agent cost report may count the same tokens twice

Token usage fields can overlap. Partition the billable categories before using an agent cost report to choose a model or approve an evaluation budget.

Judge the same frozen answers in AB and BA order, map positions back to answer identities, then retain agreement, disagreement and unknown outcomes.

Your LLM judge picked a winner. Swap the answers before you trust it.

A pairwise preference can change when the answers change places. Measure that instability before using a judge’s verdict to approve an agent release.

Four synthetic tasks run three times each: all four pass at least once, but only task A passes every trial. First-trial success is two out of four.

Your agent passed on the third try. What did the first two cost?

One successful attempt can hide an unreliable agent. Count every scheduled trial and evaluate the retry policy that users will actually experience.

Development contains the refund policy family; the held-out test contains the travel policy family. Authorised source retrieval remains allowed at test time.

Your RAG test set has new questions. Does it have new evidence?

A held-out question can still be a close relative of a development example. Use source lineage to define what your RAG evaluation actually tests.

Synthetic eval: the overall pass rate rises from 91% to 93.5%, but boundary passes fall from 100 out of 100 to 80 out of 100. The release is blocked.

Your agent's eval score improved. Its permission checks got worse.

A synthetic comparison improves from 91% to 93.5% while boundary checks deteriorate. Separate release gates keep routine successes from cancelling dangerous failures.

Both fresh and cached RAG answers pass a current authorization check before release.

Your RAG cache outlives the user's permissions

Retrieval filters cannot protect an answer that bypasses retrieval. A runnable Python fixture demonstrates the revoked-access cache failure and a conservative release gate.

Follow every inventory page, require completion evidence, then release the planning input. Stop on a timeout, cursor loop or missing evidence.

Claude Code read one page and planned for the whole inventory

A successful MCP read can still be incomplete. Require pagination and snapshot evidence before an agent uses missing items to justify a change.

Worker A holds epoch 41 and pauses. The authority grants epoch 42 to worker B. The resource accepts B and rejects the resumed worker A.

Claude Code restarted. Its old MCP worker kept writing.

A replacement worker does not silence the old one. Fence stale MCP writes at the resource boundary before a resumed job overwrites newer work.