Evals

A Claude Code cost envelope that prices tool calls before execution, reserves budget at dispatch, records actual spend, and stops over-budget paths

Make Claude Code price the tool path before it starts

Token limits miss the expensive part of many Claude Code runs. A cost envelope prices search, MCP calls, evals, and retries before the agent can spend them.

A Claude Code MCP permission delta showing one intended staging index allowed while production access, destructive DDL, another database, and role changes remain denied

Test the permission delta before widening Claude Code MCP access

A broader MCP grant may unblock one Claude Code task while authorizing unrelated production actions. Test what became possible and what must remain forbidden.

A Claude Code denial-path eval blocks a plausible production database write because the run has read-only permission

Test the Claude Code tool call that must be refused

A refusal test can pass because the tool is broken. Pair denied calls with permitted controls and adapter faults before trusting an MCP permission gate.

Diagram showing a bad Claude Code run becoming a replay case, an eval, a control change, and a safer next run

Claude Code evals should start with the run that scared you

The best Claude Code eval is not a tidy benchmark. It is the uncomfortable run your team does not want to repeat, captured as a replayable production control.

Diagram showing the operating gap between an AI POC and production AI

From AI POC to production: the part teams keep skipping

The AI POC is not the hard part anymore. The hard part is turning a promising demo into a service with ownership, evals, traces, cost controls, and a rollback path.

Diagram showing metric-only LLM observability versus a replayable production AI trace

LLM observability is not a dashboard. It is a replayable trail.

A latency chart will not explain why an AI answer was wrong. Production LLM systems need traces, sources, tool calls, prompt versions, eval results, and human decisions.

Diagram showing a Claude Code run stopping after repeated failures and producing a review packet instead of looping blindly

Claude Code needs a stop rule before more autonomy

Claude Code gets risky when a failed run keeps retrying without a stop rule. Use failure budgets, review packets, evals, and rollback notes before giving agents more autonomy.

Diagram showing a Claude Code team adoption runbook with task contract, scoped permissions, review packet, evals, and rollback

Claude Code team adoption needs a seatbelt runbook

Claude Code gets risky when teams roll it out through enthusiasm instead of a runbook. Start with task contracts, scoped permissions, review packets, evals, and rollback before widening autonomy.

Claude Code evaluation loop showing capture, reduce, test, and change steps for failed agent runs

Claude Code Evals Should Start With Bad Runs

Production Claude Code evals should not begin with abstract benchmarks. Start with the agent runs that scared you, reduce them into replayable cases, and use them to tune permissions, prompts, tools, and review gates.