Your agent got more accurate by answering fewer questions

An agent’s accuracy can rise while the service completes less work. Change its acceptance threshold, discard the rejected requests from the accuracy denominator, and the dashboard can reward a growing review queue.

I would require coverage beside every accuracy figure for an agent that can abstain. A release decision also needs the number of requests sent elsewhere and evidence that the receiving service can handle them.

Abstention is useful when an agent lacks evidence or authority. A refusal can prevent a bad answer. The accounting problem starts when a team treats that refusal as if the request never arrived.

Keep the rejected requests in the report

Consider a synthetic evaluation of 1,000 requests with adjudicated outcomes. Under a broad-coverage policy, the agent answers 900 and gets 45 of those answers wrong. Under a cautious policy, it answers 400 and gets four wrong. These are illustrative counts, not observations from a deployed service.

The cautious policy delivers fewer wrong answers in this fixture. It also leaves 600 requests for a person, another system or an explicit unresolved outcome. Whether that is an improvement depends on the harm of a wrong answer, the purpose of the service and the capacity of the next route.

The definitions are small enough to put in the release report:

• Coverage: answered requests divided by eligible requests.
• Selective risk: wrong answered requests divided by answered requests.
• Escalation volume: eligible requests minus answered requests.

Define eligibility before running the evaluation. An authorization refusal may be a correct completed decision in an access-control service. A support assistant unable to answer an eligible question has left work outstanding. Combining those outcomes under one generic refusal label makes both services harder to evaluate.

If the agent answers nothing, coverage is zero and answered-only risk is undefined. Reporting zero risk in that case gives an empty service a perfect-looking score.

Separate a routing score from a probability

Selective classification studies this trade-off between risk on accepted predictions and coverage. Geifman and El-Yaniv’s paper applies a reject option to deep neural-network classification. Its reported guarantees belong to its method and assumptions. They do not establish a guarantee for an LLM that writes “confidence: 0.9” beside an answer.

An agent may route requests using retrieval evidence, a verifier score or a task-specific acceptance rule. Those signals can help rank requests without being calibrated probabilities of correctness. Treat the threshold as a policy to evaluate. Do not turn the score’s decimal format into a reliability claim.

Use a development set to select candidate thresholds, then evaluate the selected policy on a separate held-out set. Sweeping thresholds on the final test set until one passes a target makes that set part of the selection process. Preserve a fresh test set for the release decision.

For each threshold, retain the accepted count and the wrong accepted count. Plot risk against coverage rather than comparing two accuracy figures at unrelated coverage levels. Inspect request types separately: a policy may retain routine lookups while sending almost every difficult case to review. That can be a sensible service design, but the report should say which work the agent now performs.

Test the arithmetic and the empty case

The downloadable Python fixture reproduces the table and tests the zero-answer case. It uses only the Python standard library.

def report(total, answered, wrong):
    counts = (total, answered, wrong)
    if any(type(n) is not int for n in counts):
        raise ValueError("counts must be integers")
    if not 0 <= wrong <= answered <= total or total == 0:
        raise ValueError("require 0 <= wrong <= answered <= total > 0")
    return {
        "coverage": answered / total,
        "selective_risk": wrong / answered if answered else None,
        "escalations": total - answered,
    }

This calculator checks accounting. It does not validate labels, estimate uncertainty or decide whether a threshold is safe. Small accepted sets can make a low observed error rate look more convincing than the evidence permits; the zero-failure evaluation discussion explains that limitation for a particularly tempting case.

Keep missing labels separate from abstentions. If a delivered answer has not been adjudicated, its correctness is unknown. Silently removing it from the accepted set changes the population behind the reported risk. Record the label-completion rate and investigate whether difficult cases are disproportionately waiting for review.

Give escalations an owner and an outcome

A successful handoff needs more than a queue insertion. Record when review starts, whether a reviewer resolves the request and how long the user waits. For tools that change external state, preserve enough context for the reviewer to distinguish a proposed action from one already attempted.

Keep agent-only accuracy and end-to-end service outcomes as separate measures. A human correction can rescue the user outcome without making the agent’s original answer correct. Conversely, a correct abstention can still produce a poor service outcome if nobody processes the escalation.

Before enabling a cautious threshold, replay its expected arrival volume against the review service’s measured capacity. If the queue cannot absorb the work, specify what the user receives when review is unavailable. Put the queue owner, waiting-time target and unresolved-request count beside the risk-coverage result in the release record.