AI Agent Evaluation: How to Prove Your Agent Works

AI agent evaluation is the practice of measuring whether an agent completes real tasks correctly, reliably, and inside a cost budget, using evidence rather than the impression a demo leaves in the room. It is also the question almost nobody in the current agent boom can answer about their own system. Teams ship agents weekly. Ask what their task success rate is, how it moved after the last prompt change, or how often the agent takes a wrong step and quietly recovers, and the conversation goes somewhere else.

The silence has a cause. An agent is a language model in a loop with tools and a goal, and the loop is exactly what breaks conventional evaluation. A single model call has one input and one output, so you grade the output. An agent produces a trajectory: it decides, calls a tool, reads what came back, decides again. The path can be wrong while the final answer looks right, and the answer can be wrong while every individual step looked reasonable. Grading the last message throws away most of the evidence about what happened.

What follows is a practitioner's guide to the measurement itself: the metrics and where each one lies to you, the reliability collapse that single-run scores never show, eval sets and how they decay, LLM judges and their measured biases, trajectories, cost, and what to log so a failure at 3am is diagnosable. Guardrails and human oversight close the piece, as the context evaluation operates in.

The short version

• AI agent evaluation grades a trajectory, not an output. An agent can reach the right answer by a route you would never approve, and outcome-only grading cannot see it.
• Two numbers do most of the work and constantly disagree: task success rate and step accuracy. Either alone can be gamed; the gap between them is the diagnosis.
• Single-run scores flatter agents badly. On tau-bench, Sierra reported the GPT-4o agent falling to roughly 25% at pass^8, which it describes as a 60% drop from the same agent's pass^1 score on the same tasks.
• LLM-as-judge reaches over 80% agreement with humans and carries measured biases: GPT-4 held its verdict only 65.0% of the time when two answers swapped position.
• Eval sets decay. When OpenAI had 93 developers screen SWE-bench, 68.3% of samples were filtered out as underspecified or unfairly tested. Public benchmarks are not exempt, and neither is yours.
• Cost and latency are quality metrics, not finance metrics. Anthropic found token usage alone explained 80% of the performance variance in its browsing evaluations, and that multi-agent systems use around 15 times the tokens of a chat.

What AI agent evaluation is

AI agent evaluation is the set of practices that answer one question with evidence: does this agent do the job, every time, at a price you can defend. It sits on three legs. An offline eval set of real tasks with checkable definitions of done. Automated grading that runs without a human present. And a sampled stream of production traces reviewed by people, feeding real failures back into the set.

The distinction from model evaluation matters more than it sounds. Benchmarks like MMLU grade a model's answer to a question. Agent evaluation grades a system that took actions in the world. If an AI agent queries a database, writes a record, and sends a message, the record and the message are part of the result, and any evaluation that only reads the final chat turn is grading a fraction of what happened. This is why agent evals check end state rather than text: the database matches the goal state, the file compiles, the ticket is in the right queue.

It is also why the demo is such a poor proxy. A demo grades whether the agent can succeed once, on a path someone chose, in front of people who want it to work. Evaluation grades whether it succeeds acceptably across inputs nobody scripted, repeatedly, when nobody is watching. The distance between those two sentences is where most agent budgets go.

A demo proves an agent can do the task once. Evaluation proves it does the task correctly, at a cost you can live with, on the twentieth run as well as the first. Almost all the engineering, and all the risk, lives in the gap between those two claims.

Why AI agent evaluation is harder than model evaluation

Three properties of agents break the assumptions that ordinary evaluation rests on, and each one has to be handled explicitly.

There is no single correct path. For most real tasks, several routes reach an acceptable outcome. Anthropic's evaluation guidance is blunt about the consequence: agents regularly find valid approaches that eval designers did not anticipate, so scoring against a fixed reference sequence punishes behaviour that was fine. Any step-level metric has to tolerate legitimate variation or it will report failures that are not failures.

The system is non-deterministic in a way that compounds. A model call has some variance. An agent chains many calls, and each one conditions the next, so small differences early produce entirely different trajectories later. Run the same task twice and you can get success and failure from an unchanged system. This is why a single-run score is close to meaningless as a production signal, a point the reliability section makes with numbers.

Failure is silent by default. A model that does not know something often says so. An agent that picks the wrong tool usually gets a plausible result back, incorporates it, and continues confidently. Anthropic's own note on the failure mode is that autonomy brings higher cost and the potential for compounding errors. Without instrumentation, the first signal is a customer complaint.

Set against those, the benchmark evidence on how far agents are from reliable is sobering, and worth carrying as calibration. On GAIA, a benchmark of real-world assistant tasks, human respondents score 92% while GPT-4 equipped with plugins scores 15%. On WebArena, which measures end-to-end tasks in realistic web environments, the paper's best GPT-4-based agent reaches an end-to-end success rate of 14.41% against human performance of 78.24%. Neither number means agents are useless, and both mean that assuming an agent works because it looked good is not a defensible position.

The commercial consequence is already visible. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. McKinsey found 88% of organizations using AI somewhere, but only about 23% scaling an agentic system. Unclear business value is what you get when nobody can measure the thing.

The core AI agent evaluation metrics: task success versus step accuracy

Six metrics carry most of the load, and the discipline is knowing what each one hides.

Task success rate: the headline agent evaluation metric

Task success rate is the share of eval tasks the agent completed acceptably. It is the number to lead with, because it is the one a business can act on, and it is also the number most easily inflated. Success has to be defined as a checkable condition rather than a judgement call: the refund is issued and the ledger balances, the pull request passes the existing test suite, the summary contains the five facts the task requires. If success is defined as "the answer looked good," you have built a mood ring.

Report it segmented, never as one figure. Split by task difficulty, by input type, by whether a human intervened. A flat 78% across a mixed set can hide 95% on the routine half and 60% on exactly the cases that generate complaints.

Step accuracy: where the trajectory shows its work

Step accuracy asks what fraction of the agent's decisions and tool calls were correct. It exists because the two numbers disagree, and the disagreement is diagnostic rather than noise.

High step accuracy with low task success usually means an agent that is locally competent and globally lost. Every call is well-formed; the plan is wrong, or it stops one step short. High task success with low step accuracy means it is thrashing to the right answer, which costs more, takes longer, and will break the moment a tool signature changes. Neither pattern is visible if you report only one of the numbers, which is why serious agent evaluation reports both.

End-to-end versus component-level agent evaluation

Every agent evaluation happens at one of two altitudes, and teams that work at only one of them get stuck.

End-to-end evaluation treats the agent as a black box. A task goes in, the world changes, and you check whether it changed correctly. This altitude maps to business value, and it is the only one that catches emergent failures where every component behaved and the system still did the wrong thing.

Component-level evaluation grades the parts in isolation. Does the router pick the right sub-agent, does retrieval return passages that actually contain the answer, does tool selection choose correctly given a fixed context, does the summarizer preserve the facts it was given. These are cheap and deterministic enough to run on every commit.

You need both, because they answer different questions. End-to-end tells you whether the agent works. Component-level tells you which part to fix when it does not, and it is the only practical way to debug a ten-step failure without re-running the whole trajectory a hundred times. The common trap is a team holding nothing but an end-to-end number, watching success drop three points after a change, with no way to tell which of six components moved.

The same split runs along the time axis. Offline evaluation runs a fixed set on demand, before release. Online evaluation grades live traffic after the fact against whatever signals exist: user corrections, escalation rates, downstream outcomes. Offline catches the regressions you can anticipate. Online catches the drift you cannot, and it supplies the raw material the offline set is built from.

Reliability: why pass^1 flatters and pass^k tells the truth

A single successful run tells you the agent can do the task. Production needs to know it will do the task, and the difference between those is measured by running the same task repeatedly.

Sierra's tau-bench formalized this with pass^k, which is the fraction of tasks an agent solves on all k independent runs rather than on its best attempt. It is deliberately harsh, and the results were the most useful embarrassment the field has had. In Sierra's published analysis, the best agent scored roughly 50% on pass^1, and the GPT-4o agent dropped to about 25% on pass^8 in the retail domain, which Sierra characterizes as a 60% drop from its corresponding pass^1 score. Same agent, same tasks, only the requirement that it succeed every time.

The practical rule that falls out of this is short. Pick k from what the product actually promises. An internal research assistant where a human reads every output can live with pass^1, because a bad run costs a minute. An agent that issues refunds cannot, because the eighth customer is a real customer. Then report pass^k as the headline reliability number and treat pass^1 as an upper bound you will never see in production.

Offline eval sets for agents, and how they decay

An offline eval set is a fixed collection of tasks with known-good outcomes that you run the agent against on demand. It is the backbone of the whole practice, and two things about it surprise teams.

The first is how small it can be at the start. Anthropic's engineering guidance puts 20 to 50 tasks drawn from real failures as a good beginning, and its own multi-agent research system started with about 20 queries representing real usage patterns. The logic is effect size: early prompt and tool fixes move quality a great deal, and a great deal is measurable in 20 samples. Waiting until you have 500 tasks means shipping blind for a quarter.

The second is that eval sets rot, including the famous ones. When OpenAI put SWE-bench in front of 93 experienced Python developers to build SWE-bench Verified, 68.3% of the original samples were filtered out for underspecification, unfair unit tests, or other problems, leaving 500 clean tasks. The effect on measured performance was not marginal: GPT-4o's best scaffold reached 33.2% on SWE-bench Verified against 16% on the original set. Two thirds of a benchmark the whole industry quoted turned out to be measuring something other than capability.

Your set decays in three ways, and each has a maintenance answer. Tasks go stale as the product changes and old workflows stop existing, so retire them deliberately rather than letting them fail forever. The distribution drifts as users find new things to ask, so the set has to absorb production samples continuously. And the agent overfits, because every fix is aimed at the set, so hold back a portion of tasks the team never sees during development and only run before release. Without a holdout you eventually optimize the exam, not the skill.

LLM-as-judge in AI agent evaluation, and where it misleads

Most agent output cannot be graded by code. Was the research summary accurate. Was the support reply appropriate. Was the plan sensible. So the field reaches for a second model as the grader, and the technique earns its place: Zheng et al. found GPT-4 as judge reached over 80% agreement with human preferences, which the paper notes is the same level of agreement humans reach with each other.

The same paper measured the failure modes, which is what makes it usable rather than merely encouraging.

Read that table as an instrument specification. A GPT-4 judge changed its mind 35% of the time purely because the answers moved position. Weaker judges were fooled over 90% of the time by an answer padded with repetition and no new information. Judges favour their own output. None of this makes the technique unusable; all of it makes an uncalibrated judge score untrustworthy as a number.

The working practice is four rules. Run every comparison in both orders and average, which converts position bias into noise rather than a systematic tilt. Score against an explicit rubric with named dimensions rather than a general quality question, the way Anthropic's multi-agent research system graded factual accuracy, citation accuracy, completeness, source quality, and tool efficiency separately. Calibrate the judge against human labels on a sample, and keep re-calibrating, because Anthropic's guidance is explicit that judges should stay closely aligned with human experts. And never let the judge be the same model as the agent, which turns self-enhancement bias into a self-grading system.

Human review sampling: what automated agent evaluation misses

Automated evaluation scales and is blind in a specific way: it only measures what someone already thought to measure. Anthropic's account of building its multi-agent research system puts it plainly, that people testing agents find edge cases evals miss, including hallucinated answers on unusual queries, system failures, and subtle source selection biases. Every one of those is invisible to a grader looking for a known-good outcome.

Human review is how you find the unknown unknowns, and the discipline is in the sampling rather than the reading. Review a fixed small percentage of production runs, chosen randomly, so the sample represents reality. Then oversample from the tails: runs far longer than the median, runs where a guardrail fired, runs the user abandoned, runs where the judge and the outcome check disagreed. That second stream is where the interesting failures live, and it is nearly free because the traces already exist.

The output of a review session is not a score. It is new eval tasks. Every genuine failure a human finds becomes a case in the offline set, which is what stops the same bug from shipping twice and what keeps the set tracking reality instead of drifting away from it.

Trajectory analysis: evaluating the path, not just the answer

A trajectory is the ordered record of what the agent did: each model call, each tool invocation with its arguments, each result, each decision to continue or stop. Trajectory analysis grades that record, and it catches a category of failure outcome grading cannot see. Right answer reached through a tool that should never have been touched. Right answer after eleven steps that needed three. Right answer produced by an agent that wrote to production and then read its own write back as confirmation.

Google Cloud's Vertex AI ships this as production metrics, with trajectory exact match, precision, and recall comparing the calls an agent made against a reference path. Precision and recall are the useful pair: precision asks how many of the agent's calls belonged in the solution, recall asks how many of the necessary calls it made.

The counterweight is real and worth stating clearly. Anthropic's evaluation guidance recommends grading what the agent produced rather than the path it took, precisely because agents find valid approaches designers did not anticipate, and rigid step matching penalises them for being clever. Both positions are correct at different jobs. Grade outcomes to decide whether the agent is good. Read trajectories to understand why it is good or bad, and to enforce the small number of paths that are genuinely forbidden regardless of outcome.

In practice that means three trajectory checks worth automating: tool-call validity, whether arguments matched the schema and the result was usable; step count against a per-task budget, which catches thrashing before the bill does; and a forbidden-action check, which fails a run outright if it touched something outside its permitted scope even when the answer was correct. This is where the difference between monitoring and observability stops being semantic: monitoring tells you a run failed, a trajectory tells you which step did it.

Regression suites: AI agent evaluation that runs on every change

An agent has more moving parts than a normal service, and every one can silently change behaviour. A prompt edit, a tool whose response shape changed, a reindexed retrieval store, or a model version the provider bumped on its own schedule. Any of these degrades quality without producing a single error, and none is caught by tests that assert on structure rather than outcomes.

The answer is ordinary engineering applied to an unusual system. The eval set becomes a regression suite that runs on every change to the agent, in CI, exactly as Anthropic recommends for pre-launch and model upgrades. The gate is a threshold on task success and pass^k rather than a pass or fail on individual cases, because a non-deterministic system will always fail some run somewhere.

Three details make the difference between a suite people trust and one they route around. Pin the model version explicitly, so a provider-side update shows up as a deliberate change with an eval run attached rather than as a mysterious Tuesday. Run enough trials per task that the threshold is not noise; a suite that flaps red will be ignored within two weeks. And treat cost as a gated metric alongside quality, because a prompt change that lifts success by two points while doubling spend is a regression in every sense that matters to the business.

Cost and latency as first-class agent evaluation metrics

Cost and latency are quality metrics for agents, and treating them as a finance concern to review later is how projects get cancelled for unclear business value.

The magnitudes are published. Anthropic reports that agents typically use about 4 times more tokens than chat interactions, and multi-agent systems about 15 times more. That multiplier applies to every run, including the failures, which is why cost per successful task is the honest denominator. An agent that costs half as much per call but fails a third of the time is more expensive, and the naive per-call number will never show it.

More interesting is that spend is a quality signal in its own right. In Anthropic's analysis of its browsing agent, token usage by itself explained 80% of the performance variance, with three factors together explaining 95%. Tokens are how an agent thinks, so a run that used far fewer than usual often did less work than the task required, and a run that used far more was probably lost. Both are worth alerting on before a human notices the output is wrong.

Latency needs the same treatment, measured at the percentile users feel rather than the mean. A step-heavy agent has a long tail by construction, and the tail is where abandonment happens. Anthropic states the trade plainly: agentic systems often trade latency and cost for better task performance, and you should decide when that trade makes sense rather than discovering it in an invoice. Where the extra steps do not buy accuracy, a narrower deterministic workflow is the better system, which is most of the value in an honest read on AI ROI.

What to log so agent failures stay diagnosable

Everything above depends on the trace existing. If the run was not recorded in enough detail, a failure report is a guess, and the eval set never learns from production.

There is now a standard worth adopting rather than inventing your own. The OpenTelemetry GenAI semantic conventions define the span shape for exactly this: an invoke_agent span at the top with chat spans for each model call and execute_tool spans for each tool invocation beneath it, carrying gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for spend and gen_ai.response.finish_reasons for why generation stopped. Instrumenting to the convention means your traces work with whatever tooling you pick later, and it settles a hundred small naming arguments before they start.

On top of the standard span data, five fields make the difference between a debuggable run and a mystery. The full tool call, with arguments and the raw result, because a wrong argument is the single most common root cause and it is invisible if you log only the tool name. The task or session identifier, so all steps of one job stitch together. A version stamp covering prompt, model, and tool schema, so you can answer which change broke it. The outcome verdict and grader used, so a production trace can be replayed into the eval set without re-adjudication. And the guardrail events, every cap hit, permission denial, and approval request, because those are the tail you want to oversample for human review.

One caution worth stating: traces contain user data by construction. The convention allows content capture and it should be a deliberate decision with redaction on the fields that need it, not an accident of enabling verbose logging in a regulated system.

Guardrails and human-in-the-loop: the context evaluation sits in

Evaluation tells you how often the agent is wrong. Guardrails and human oversight decide what happens when it is, and neither substitutes for the other. This is what the journey from AI pilot to production demands of every use case, with extra weight when the actor is autonomous.

Guardrails bound the blast radius. Cap the number of steps so a confused agent cannot loop indefinitely. Cap spend per task and per day. Scope every credential to the minimum the task needs. Validate the arguments going into a tool and sanity-check what comes back. An agent with real tools and no limits is an incident waiting for a trigger, and evaluation on its own will not stop it, because evaluation runs after the fact.

Human-in-the-loop covers the actions where being wrong is expensive or permanent: moving money, deleting records, sending the message that cannot be recalled. The agent proposes; a person approves the ones that matter.

Between them, these two answer the question teams find hardest, which is whether an agent is ready to ship. There is no universal success rate that qualifies it. The threshold is set by what a failure costs and by what catches it before a user does, so an agent whose output a human reviews can launch at a number that would be reckless for one issuing refunds unattended. Readiness is a statement about the whole system, and the evaluation record is what makes it defensible, because you can show what the agent was tested against and how often it passed. We build to that bar on an eIDAS-qualified e-signature platform, a digital-banking platform, and supply-chain systems carrying real liability, and it is the operating model we are compounding toward 2028.

Standing up this whole layer, the eval harness, the tracing, the caps, and the approval gates, is most of what our AI agent development practice actually delivers. The loop is a weekend. The measurement around it is the job.

How to start evaluating an AI agent

If you have an agent and no evaluation, the order below gets you to a defensible position in weeks rather than quarters. Each step produces something usable on its own.

1. Collect 20 to 50 real tasks. Pull them from support tickets, logs, and the failures people already complain about. Invented tasks measure an imagined product. This is the step teams skip because it feels unglamorous, and it is the step everything else rests on.
2. Define done as a checkable condition. For each task, write what the world should look like afterwards: the record state, the file that compiles, the facts that must appear. If you cannot express success as a check, you cannot evaluate that task, and that is worth knowing before you build the agent.
3. Grade with code where you can and a judge where you cannot. Objective outcomes get an assertion. Subjective ones get a rubric-based judge, run in both orders, calibrated against human labels on a sample.
4. Measure reliability, not just success. Run each task multiple times and report pass^k at a k your product actually needs. This is the number that will surprise you, and it is the one production depends on.
5. Put the set in CI. Every prompt, model, and tool change runs it. Gate on a threshold, pin the model version, and gate cost alongside quality.
6. Trace production and sample it. Instrument to the OpenTelemetry conventions, review a small random slice plus the tails, and feed every real failure back into the set as a new task.

The thread through all six is that AI agent evaluation is a loop, not a phase. The set that measured last quarter's agent will quietly stop representing this quarter's users, the judge that was calibrated in March will drift, and the benchmark everyone quotes may turn out to be two thirds noise. What separates teams whose agents get better from teams whose agents merely get shipped is that the first group can produce a number, defend how it was measured, and show it moving. Where a single agent fits a wider plan is the subject of building an enterprise AI strategy, which functions people actually deploy them in is covered in our guide to AI agents for business, and what working deployments look like in the wild is the subject of real AI agent examples.