AI | Thomas Talks AI

Constructed audiences checked against https://pay.example/mcp. The exact value is allowed. A different server and a longer path are denied. A token naming two servers passes membership and fails a single-audience policy.

Azure AD Graph accepted a token issued for a different tenant

Dirk-jan Mollema changed the tenant on an impersonation token whose signed Actor token named his lab. Azure AD Graph accepted it. CVE-2025-55241 followed. The MCP lesson is an exact audience check on constructed claims.

Two synthetic comparisons improve from 80 to 90 passes out of 100. A gains ten cases and loses none. B gains twenty and loses ten.

Your agent gained ten passes. Which cases did it lose?

Two synthetic agent comparisons both improve from 80 to 90 passes. Pairing the outcomes reveals ten regressions in one comparison and none in the other.

Synthetic worker latency fixture: worker A has 100 requests at one second, worker B has 900 at ten seconds; the combined nearest-rank p95 is ten seconds.

Averaging your agents' p95 latencies gives you the wrong percentile

Worker percentiles discard the distribution you need for a service percentile. A reproducible fixture shows the error and a histogram query shows what to aggregate instead.

Two synthetic agent policies: broad coverage answers 900 of 1000 requests; cautious coverage answers 400, leaving 600 for another route.

Your agent got more accurate by answering fewer questions

An agent can raise its answer accuracy by refusing more work. Report the accepted answers, the rejected requests and the review queue together before changing its threshold.

Zero observed failures with 95% one-sided upper bounds: 30 trials gives 9.50%, 100 gives 2.95%, and 300 gives 0.99%. These bounds require independent trials with a fixed failure probability.

Your agent passed every test. How much failure did you rule out?

A clean evaluation needs a sample-size argument. Calculate what zero failures supports, and separate statistical uncertainty from missing or correlated evidence.

Synthetic example: 100 runs contain five failures. One retained subset has zero failures in ten traces; a failure-enriched subset has five in fifteen. Neither raw subset rate describes all runs.

Your agent failure rate depends on which traces you kept

A failure-enriched trace store is useful for investigation. Its raw failure percentage is a poor release metric. Keep an independent outcome denominator.

Partition 10,000 input tokens into ordinary input, cache reads and cache writes. Count 1,000 output tokens once, including their 700 reasoning tokens.

Your agent cost report may count the same tokens twice

Token usage fields can overlap. Partition the billable categories before using an agent cost report to choose a model or approve an evaluation budget.

Judge the same frozen answers in AB and BA order, map positions back to answer identities, then retain agreement, disagreement and unknown outcomes.

Your LLM judge picked a winner. Swap the answers before you trust it.

A pairwise preference can change when the answers change places. Measure that instability before using a judge’s verdict to approve an agent release.

Four synthetic tasks run three times each: all four pass at least once, but only task A passes every trial. First-trial success is two out of four.

Your agent passed on the third try. What did the first two cost?

One successful attempt can hide an unreliable agent. Count every scheduled trial and evaluate the retry policy that users will actually experience.

Development contains the refund policy family; the held-out test contains the travel policy family. Authorised source retrieval remains allowed at test time.

Your RAG test set has new questions. Does it have new evidence?

A held-out question can still be a close relative of a development example. Use source lineage to define what your RAG evaluation actually tests.