Evals

Synthetic task-mix reversal: routine and complex pass rates both fall, while the observed total rises from 45% to 75%. Fixed baseline weights give 45% to 35%.

Your agent got worse at every task group. Its total pass rate went up.

A synthetic agent’s observed pass rate rises from 45% to 75%, despite regressions in both task groups. Fixed reference weights expose the traffic-mix reversal.

For unit vectors, cosine similarity of at least 0.8 corresponds to cosine distance at most 0.2 and squared L2 distance at most 0.4.

Your RAG ranking survived the migration. Its relevance threshold did not.

A retrieval migration can preserve the top results while changing which documents your answer gate accepts. Test the score contract as well as the ranking.

Synthetic cohort of 100 arrivals: 90 successes at 100 ms and ten timeouts at 2000 ms. Success-only p95 is 100 ms, but only 90 percent succeed within 500 ms.

Your agent's p95 looks fast because the timeouts disappeared

Ninety fast successes and ten timeouts produce a flattering success-only p95. Account for the full arrival cohort before declaring an agent service fast enough.

Two synthetic comparisons improve from 80 to 90 passes out of 100. A gains ten cases and loses none. B gains twenty and loses ten.

Your agent gained ten passes. Which cases did it lose?

Two synthetic agent comparisons both improve from 80 to 90 passes. Pairing the outcomes reveals ten regressions in one comparison and none in the other.

Exact-ranking fixture: selecting three candidates before the permission filter returns none; filtering first returns three allowed documents, including both relevant answers.

Your RAG filter blocks leaks and still hides the right answer

An empty result can mean the candidate budget ran out before the permission filter found useful evidence. Test authorised recall separately from disclosure.

Follow every inventory page, require completion evidence, then release the planning input. Stop on a timeout, cursor loop or missing evidence.

Claude Code read one page and planned for the whole inventory

A successful MCP read can still be incomplete. Require pagination and snapshot evidence before an agent uses missing items to justify a change.

A review packet has passing tests but a missing MCP write receipt. Further agent writes are blocked while an authorised operator reconciles the outcome.

Claude Code review packets: unknown is not a pass

Passing tests cannot settle a missing MCP write receipt. Give unresolved effects a status that the approval gate cannot ignore.

An agent writes revision 41, an engineer writes revision 42, and conditional recovery refuses to overwrite the engineer's repair

Claude Code rollback must not erase the next engineer's fix

A saved snapshot can undo the agent’s change and somebody else’s repair. Make recovery conditional on the revision the agent wrote.

One Claude Code MCP call fans out to paid workers while a shared cost capability blocks work beyond the 25 dollar limit

One Claude Code MCP call spent the whole agent budget

The Claude Code run had a $25 limit and made one MCP call. Hidden fan-out behind the server produced a $286.40 bill.

A Claude Code incident replay forks into a faithful path using captured MCP evidence and a misleading path that calls today's live tools

Your Claude Code incident replay used today's tool output

The same prompt and repository commit produced a clean replay because an MCP tool returned newer evidence. The team had rerun the task, not reproduced the incident.