AI Agent Evaluation: What to Measure Before You Ship One
The demo works. Somebody on the team has run it twenty times, it did the right thing nineteen of those, and now the question is whether it can face customers. That question cannot be answered by looking at outputs, which is what most teams try first.
Evaluating a language model asks whether the text was good. Evaluating an agent asks whether it did the right things, in a sensible order, using the right tools, and then stopped. Those are different questions, and the second one can fail while every individual response looks fine.
An agent that answers a billing query correctly after calling the refund tool twice has produced a good output and a bad trajectory. An agent that gets the right answer by looping eleven times through a search it should have run once has produced a good output and an unaffordable one. Neither failure is visible in the final response, which is why output-only testing gives teams false confidence right up until launch.
The second structural problem is that there is often no single correct answer. For a given request, several tool sequences may be legitimate. Grading against one golden trajectory punishes correct behaviour that happens to differ, so the useful question is usually whether the agent satisfied a set of conditions rather than whether it matched a script.
Three things decide whether your evaluation is worth anything. Does it test trajectories as well as outputs? Does it run on real traffic rather than invented examples? And does a failing eval actually block a deploy? The third is where most efforts quietly die.
Evaluate an agent on its trajectory, not only its final answer: which tools it called, in what order, and whether it stopped. Start with fifty real traces from production or a pilot rather than invented test cases, hold back about a fifth you never tune against, and grow the set from every failure.
The instinct is to sit down and write test cases. It produces a test set that reflects what the team imagined users would do, which is reliably not what users do.
Take fifty real traces from production or pilot traffic and make those your first dataset. They carry the ambiguity, the typos, the half-formed requests and the weird edge cases that no one invents. If you have no traffic yet, run an internal pilot specifically to generate them before you build the eval.
Grow the set weekly from failures. Every production incident becomes a case. This is the single habit that separates eval suites that get better from ones that rot, and it costs almost nothing once the plumbing exists.
Hold back a portion you never tune against. Teams that iterate against their whole dataset end up with an agent that performs beautifully on exactly those cases and no better in the wild. Reserving roughly a fifth as a test set you only look at before a release is a cheap guard against that.
Cover the boring cases deliberately. Most eval sets over-represent the interesting failures and under-represent the ninety per cent of traffic that is routine, which means a regression in the common path can pass unnoticed.
Redact before you store. Real traces carry real customer data, and an eval dataset is a copy of production that lives in a repository and gets shared with everyone who works on the agent. Strip or pseudonymise identifiers at capture time rather than promising to do it later.
Label what good looks like at the time you capture the trace, while someone remembers. Retroactive labelling six weeks later is slow and inconsistent.
The trajectory is the sequence of decisions: which tools were called, with what arguments, in what order, and when the agent decided it was finished.
Tool selection is the first thing to assert. Did the agent call the tool the task required? Calling a search when it should have called the account lookup is a failure even when the final text happens to be right, because it will not be right next time.
Argument correctness matters as much as tool choice. An agent that calls the right endpoint with a malformed date or the wrong customer identifier has failed in a way that output grading will miss entirely, especially if the tool fails gracefully.
Order matters where actions have side effects. Checking eligibility before issuing a refund is not the same as issuing a refund and then checking.
Termination is the most underrated assertion. Agents loop. They retry a failing tool indefinitely, or re-plan forever, or stop early on a partially completed task. Assert a maximum step count and assert that the agent reached a terminal state for the right reason.
Assert on conditions rather than exact sequences wherever several paths are valid: the refund tool was called exactly once, the eligibility check preceded it, no destructive tool was called, the step count stayed under twelve. That style survives legitimate variation and still catches real failures.
Agents fail in a small number of recognisable ways, and a suite that covers these catches most of what reaches production.
Looping. The agent retries a failing tool, or re-plans the same step, or alternates between two tools indefinitely. It is the most common expensive failure and the easiest to assert against with a step cap.
Premature stopping. The agent declares success on a partially completed task, usually because the final response reads plausibly. This one is invisible to output grading by construction, since the output is exactly the problem.
Hallucinated tool arguments. The agent invents an identifier, a date or a parameter the user never supplied. Strict schema validation on tool inputs catches most of it, and asserting that every argument traces back to the conversation or a prior tool result catches the rest.
Context loss over long tasks. Twenty steps in, the agent has forgotten a constraint stated at the beginning. Write cases that state a constraint early and check compliance late; most agents that pass short tests fail these.
Silent tool failure. A tool returns an error, the agent treats it as data, and proceeds confidently on nothing. Test every tool's failure path explicitly, because the happy path is the one everyone already tried.
Over-eager action. The agent takes a consequential step the user did not ask for, which is the failure that generates incidents rather than complaints. Assert that destructive tools are never called without an explicit instruction in the trace.
Both layers are needed and they answer different questions.
End-to-end evaluation runs the whole agent against a case and grades the result. It is the number you report and the one that correlates with user experience. It is also nearly useless for debugging, because a failure tells you the agent got it wrong without telling you where.
Component evaluation isolates the parts: the planner, the tool-selection step, the retrieval layer, the final formatting. When the end-to-end number drops, component evals tell you which stage moved.
Retrieval deserves its own evaluation if the agent uses it. A large share of agent failures are retrieval failures wearing a reasoning costume — the agent reasoned perfectly over the wrong documents. Measuring retrieval separately stops you tuning a prompt to compensate for a search problem.
Run end-to-end on every change and components on failures. End-to-end is slower and more expensive; components are cheap and diagnostic. Running the full matrix on every commit is a cost most teams abandon within a month.
Much of what you want to grade is not mechanically checkable. Was the explanation accurate? Was the tone appropriate? Did it answer the actual question? Using a language model to grade these is standard practice and it is also where eval suites go wrong.
Calibrate the judge before trusting it. Have humans grade a sample, have the judge grade the same sample, and measure the agreement. If the judge agrees with your reviewers 70% of the time, your eval has a 30% error bar and should be read that way.
Judges have known biases. They tend to prefer longer answers, prefer their own style of writing, and are influenced by the order in which options are presented. None of those correlate with being right.
Prefer checkable assertions wherever one exists. "The refund tool was called exactly once" needs no judge. Reserve model grading for the genuinely subjective parts, and you will spend less and trust more.
Never let the judge be the same model under the same prompt as the agent. It shares the failure modes you are trying to detect.
Show the judge the trajectory, not only the answer, when the thing being graded is behaviour. A judge given the final text alone cannot tell you whether the agent took a sensible route to it.
Re-calibrate when you change models. A judge calibrated against one model version is not calibrated against the next one, and model updates arrive whether you plan for them or not.
Offline evaluation runs against a fixed dataset before deployment. Online evaluation watches real traffic. Teams do the first and skip the second, and the second is where the surprises live.
Offline is for regression. Its job is to tell you that today's change did not break yesterday's behaviour, which is the failure mode of prompt-driven systems: a small wording change improves one case and silently degrades four.
Online is for drift and for the unknown. Users do things your dataset does not contain, upstream models change under you, and your own data changes shape. Without online measurement these surface as complaints.
The online signals worth instrumenting from day one are completion rate, escalation or handoff rate, step count per task, tool error rate, and cost per completed task. None requires a judge and all are available from logs.
Sample traffic for human review on a schedule, a small number per week, read properly. This catches the category of problem no automated metric is looking for and is the cheapest quality intervention available.
Watch for the metric that only moves when nobody is looking. Escalation rate climbing by two points a week is invisible day to day and obvious over a month, so review these on a weekly cadence rather than a dashboard nobody opens.
Feed every online failure back into the offline set, which is how the two layers compound instead of existing separately.
The quality of an eval suite is set by who decides what correct means, and that person is usually not the person building it.
Engineers write cases that test what the system does. Support staff, operations people and whoever currently does the job by hand write cases that test what the business needs, and the two sets overlap less than anyone expects.
The fastest route to a good dataset is to sit with the person the agent is replacing or assisting. They can tell you in an hour which requests are common, which are dangerous, and which look simple and are not. That conversation is worth more than a week of writing cases from the specification.
Disagreement between reviewers is information, not noise. If two people grade the same trace differently, the definition of correct is unclear, and that ambiguity is in your prompt too. Resolve it in the eval and the agent usually improves without a code change.
Keep the cases readable by non-engineers. A dataset expressed in plain language with expected conditions can be reviewed by the people who understand the domain. One buried in test code can only be reviewed by its author, which is how eval suites drift away from the business.
Review the dataset on a schedule. Products change, policies change, and a case that encoded last year's refund rule now asserts the wrong behaviour. An eval suite nobody prunes eventually blocks correct changes, which is how teams learn to ignore it.
For an agent, how much it spent and how long it took are not operational afterthoughts. They are part of whether it worked.
An agent that reaches the right answer in forty steps has failed if the task needed four, because it cost ten times as much and took ten times as long. Track steps per task and tokens per task as first-class results in the eval, not as infrastructure metrics.
Cost per completed task is the number to report upward. Cost per call is meaningless when one task may involve twenty calls, and it is the metric that makes an agent look cheap right up to the invoice.
Watch the distribution rather than the mean. Agent cost is long-tailed: most tasks are cheap and a few loop expensively. The mean hides exactly the cases that hurt, so track the 95th percentile.
Set budget assertions in the eval. Fail the build when median steps or median cost regresses beyond a threshold. This is the only reliable defence against a prompt change that improves quality by spending three times more.
An eval suite is a real piece of engineering and it is also small compared with the agent it protects.
The minimum useful version is a dataset, a runner, a set of assertions and a report, wired into continuous integration so a failing eval blocks a merge. Our rate band is published, $40 to $100 an hour by role, blending to $60 to $70 for a mixed team, and a first suite is a matter of days rather than weeks.
Buy the tracing and observability layer. Capturing, storing and browsing agent traces is a solved problem with mature tools, and building it yourself is a distraction from the part that is specific to you.
The part you must own is the dataset and the assertions, because they encode what correct means for your business and nobody can supply that.
Build it before the agent, not after. Teams that write the suite first ship slower in week one and faster from week three, because every subsequent change is verifiable. Teams that defer it accumulate an agent whose behaviour nobody can characterise, and the cost of retrofitting evaluation grows with every prompt revision.
Completion rate on fifty real traces, graded by a human, to establish a baseline nobody can argue with.
Step count and cost distribution on those same traces, because they are free to collect and immediately reveal the looping cases.
Tool-selection accuracy as a mechanical assertion, which usually surfaces more real defects in an afternoon than a week of reading outputs.
Termination behaviour under deliberately awkward inputs: a request the agent cannot fulfil, a tool that returns an error, a question outside its scope. How an agent stops is the behaviour that most often embarrasses people in production.
Then, and only then, the subjective quality grading everyone wants to start with. An agent that calls the right tools, in the right order, and knows when to stop is most of the way to being good. One that writes beautifully and loops forever is not shippable at any quality score.
It is testing whether an agent did the right things rather than only whether its final answer looked right. Language model evaluation grades generated text; agent evaluation grades behaviour — which tools were called, with what arguments, in what order, and whether the agent stopped correctly. An agent can produce a perfect answer through a wrong or wasteful trajectory, and that failure is invisible to output-only testing.
An LLM produces one output you can grade. An agent produces a sequence of decisions, any of which can be wrong while the final text still reads well. There is also frequently no single correct trajectory, since several tool sequences may be legitimate, so grading against one golden path punishes correct behaviour that differs. Agent evaluation therefore asserts conditions — the refund tool was called once, eligibility preceded it, step count stayed under twelve — rather than matching a script.
Start with about fifty real traces from production or pilot traffic rather than invented examples, because real requests carry ambiguity and malformed inputs that nobody imagines. Grow the set weekly by turning every production failure into a case, and hold back roughly a fifth that you never tune against, so you can tell whether you improved the agent or only fitted it to your dataset.
Yes, for the genuinely subjective parts, but calibrate it first. Have humans grade a sample, have the judge grade the same sample, and measure agreement; if they agree 70% of the time, your eval carries a 30% error bar. Judges are known to favour longer answers and their own writing style, neither of which correlates with correctness. Use mechanical assertions wherever one exists, and never use the same model and prompt as the agent being judged.
Offline evaluation runs against a fixed dataset before deployment and exists to catch regressions, which is the characteristic failure of prompt-driven systems where a small wording change improves one case and degrades four. Online evaluation watches real traffic and catches drift and the unknown: users doing things your dataset never contained, upstream model changes, and shifting data. You need both, and every online failure should become an offline case.
Yes, as correctness metrics rather than operational ones. An agent that reaches the right answer in forty steps when four would do has failed, because it costs and takes ten times more. Track steps and tokens per task in the eval itself, report cost per completed task rather than per call, and watch the 95th percentile rather than the mean, since agent cost is long-tailed and the mean hides the looping cases.
Buy the tracing and observability layer, since capturing and browsing agent traces is a solved problem with mature tooling. Build and own the dataset and the assertions, because they encode what correct means for your business and no vendor can supply that. Build it before the agent rather than after: teams that write the suite first are slower in week one and faster from week three, because every change afterwards is verifiable.
Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.
Share your details and we will talk soon.
Be the first to access expert strategies, actionable tips, and the trends actually shaping the digital world. No fluff - just practical insights delivered straight to your inbox.
Dive into our blog and stay ahead of the curve with expert perspectives, future-ready trends, and tech tips written for decision-makers and doers alike.
