Posts

Synthetic task-mix reversal: routine and complex pass rates both fall, while the observed total rises from 45% to 75%. Fixed baseline weights give 45% to 35%.

Your agent got worse at every task group. Its total pass rate went up.

A synthetic agent’s observed pass rate rises from 45% to 75%, despite regressions in both task groups. Fixed reference weights expose the traffic-mix reversal.

For unit vectors, cosine similarity of at least 0.8 corresponds to cosine distance at most 0.2 and squared L2 distance at most 0.4.

Your RAG ranking survived the migration. Its relevance threshold did not.

A retrieval migration can preserve the top results while changing which documents your answer gate accepts. Test the score contract as well as the ranking.

One JSON payload names staging and production. A synthetic first-wins preview shows staging, while Python's default decoder returns production.

Your tool approval says staging. The JSON also says production.

A first-wins preview and a last-wins executor can disagree about the same tool arguments. Both decoded objects can still pass the same schema.

Three layers allowing three total attempts each produce up to 27 dependency calls. Giving only the SDK three attempts reduces the fixture to three calls.

Three retry layers turn one agent request into 27 calls

An agent loop, tool adapter and SDK can each obey their retry limit while multiplying load on a failing service. Measure the calls at the dependency boundary.

Synthetic cohort of 100 arrivals: 90 successes at 100 ms and ten timeouts at 2000 ms. Success-only p95 is 100 ms, but only 90 percent succeed within 500 ms.

Your agent's p95 looks fast because the timeouts disappeared

Ninety fast successes and ten timeouts produce a flattering success-only p95. Account for the full arrival cohort before declaring an agent service fast enough.

Constructed audiences checked against https://pay.example/mcp. The exact value is allowed. A different server and a longer path are denied. A token naming two servers passes membership and fails a single-audience policy.

Azure AD Graph accepted a token issued for a different tenant

Dirk-jan Mollema changed the tenant on an impersonation token whose signed Actor token named his lab. Azure AD Graph accepted it. CVE-2025-55241 followed. The MCP lesson is an exact audience check on constructed claims.

Two synthetic comparisons improve from 80 to 90 passes out of 100. A gains ten cases and loses none. B gains twenty and loses ten.

Your agent gained ten passes. Which cases did it lose?

Two synthetic agent comparisons both improve from 80 to 90 passes. Pairing the outcomes reveals ten regressions in one comparison and none in the other.

Exact-ranking fixture: selecting three candidates before the permission filter returns none; filtering first returns three allowed documents, including both relevant answers.

Your RAG filter blocks leaks and still hides the right answer

An empty result can mean the candidate budget ran out before the permission filter found useful evidence. Test authorised recall separately from disclosure.

Synthetic worker latency fixture: worker A has 100 requests at one second, worker B has 900 at ten seconds; the combined nearest-rank p95 is ten seconds.

Averaging your agents' p95 latencies gives you the wrong percentile

Worker percentiles discard the distribution you need for a service percentile. A reproducible fixture shows the error and a histogram query shows what to aggregate instead.

Two synthetic agent policies: broad coverage answers 900 of 1000 requests; cautious coverage answers 400, leaving 600 for another route.

Your agent got more accurate by answering fewer questions

An agent can raise its answer accuracy by refusing more work. Report the accepted answers, the rejected requests and the review queue together before changing its threshold.