AI | Thomas Talks AI

NaN bypasses a reject-below-threshold check. A repaired gate validates the score first and returns INVALID before threshold comparison.

Your agent eval returned NaN. The release gate passed.

A check that rejects scores below a threshold can approve NaN. Validate the score contract before making a release decision.

In a harmless parser fixture, the argument --delete enables an option even with shell=False. An option separator keeps it as data; identifier validation rejects it.

Your agent uses shell=False. The tool still accepts an injected option.

A subprocess argument list preserves argument boundaries. The called program still decides whether each argument is data or an instruction.

A 2068-byte gzip fixture passes a 64 KiB input check but expands to 2 MiB. A bounded decoder rejects it after detecting that it exceeds a 256 KiB output limit.

A small gzip tool result can exceed your agent's output budget

A compressed-byte check does not bound decoded output. Test the decompressor’s allocation contract before passing tool results to an agent.

Your agent's 30-second timeout grew when the clock moved backwards

Use one monotonic deadline for local elapsed-time budgets. Queue waits, retries and backoff must spend the same allowance.

A lexical path check accepts allowed/docs/note.txt, but docs is a symlink outside. A narrower reader accepts one filename and opens it relative to a trusted directory descriptor.

Your agent's file path is allowed. Its symlink points outside.

An allowed path can lead to an unauthorised file. A runnable Python lab demonstrates lexical checks, a deterministic check/use race, and a bounded file reader.

Synthetic task-mix reversal: routine and complex pass rates both fall, while the observed total rises from 45% to 75%. Fixed baseline weights give 45% to 35%.

Your agent got worse at every task group. Its total pass rate went up.

A synthetic agent’s observed pass rate rises from 45% to 75%, despite regressions in both task groups. Fixed reference weights expose the traffic-mix reversal.

For unit vectors, cosine similarity of at least 0.8 corresponds to cosine distance at most 0.2 and squared L2 distance at most 0.4.

Your RAG ranking survived the migration. Its relevance threshold did not.

A retrieval migration can preserve the top results while changing which documents your answer gate accepts. Test the score contract as well as the ranking.

One JSON payload names staging and production. A synthetic first-wins preview shows staging, while Python's default decoder returns production.

Your tool approval says staging. The JSON also says production.

A first-wins preview and a last-wins executor can disagree about the same tool arguments. Both decoded objects can still pass the same schema.

Three layers allowing three total attempts each produce up to 27 dependency calls. Giving only the SDK three attempts reduces the fixture to three calls.

Three retry layers turn one agent request into 27 calls

An agent loop, tool adapter and SDK can each obey their retry limit while multiplying load on a failing service. Measure the calls at the dependency boundary.

Synthetic cohort of 100 arrivals: 90 successes at 100 ms and ten timeouts at 2000 ms. Success-only p95 is 100 ms, but only 90 percent succeed within 500 ms.

Your agent's p95 looks fast because the timeouts disappeared

Ninety fast successes and ten timeouts produce a flattering success-only p95. Account for the full arrival cohort before declaring an agent service fast enough.