How to Read an AI Vendor's Accuracy Claim

11 min read
How to Read an AI Vendor's Accuracy Claim

A vendor shows you 94% accuracy. The slide is clean, the number is large, and the only honest response is a question: 94% of what, measured by whom, on whose data?

A benchmark score is a measurement of a benchmark. It describes performance on a fixed set of tasks, chosen by somebody, under conditions that may or may not resemble yours. That is useful information. It is not a statement about what the system will do in your business, and the gap between those two things is where most disappointed AI purchases live.

The research community has already had this argument and the useful part is the definition. Dekoninck, Müller and Vechev, in work on contamination detection, propose defining contamination not as benchmark data appearing in training data but as "artificially inflated and non-generalizing benchmark performance". Under that definition a model is contaminated when its score is inflated and does not carry over to rephrased samples, to synthetic samples from the same distribution, or to other benchmarks for the same task. They report high levels of it in several widely used models.

Read that as a buyer rather than as a researcher and it becomes the only test that matters. Contamination is not a question about the vendor's ethics. It is a question about whether the number transfers. And the thing you are buying is transfer.

A second finding sharpens it. Singh and colleagues, examining how contamination is measured, conclude that "contamination may have a much larger effect than reported in recent LLM releases", and that it benefits models differently at different scales, attributing the gap to metric settings that produce many false negatives. So the published corrections are themselves probably conservative.

Three questions settle most claims. What exactly was measured? Who measured it? And what is the denominator? None of them requires you to be a machine learning engineer, and all three can be asked in a first meeting.

How do you evaluate an AI vendor's accuracy claim?

Ask what was measured, who measured it, and on what denominator. A score is performance on a fixed benchmark, and the useful definition of contamination is a score that is inflated and does not carry over to rephrased or equivalent tasks. Then stop arguing and run the vendor's system on fifty of your own real cases with known answers.

That last step ends most discussions in an afternoon. The rest of this post is what to ask before you get there, and how to build the fifty cases so the result means something.

Who measured it, and is the measurement independent

There is a difference between a number a vendor reports and a number somebody else can reproduce, and the difference is usually visible if you look for it.

The honest version of this is visible in computer-use agents. On OSWorld, which tests agents on open-ended tasks in real desktop environments, the paper that introduced the benchmark in 2024 reported that humans could accomplish over 72.36% of the tasks while the best model managed 12.24%. Read on 13 November 2026, the project's own leaderboard put its top verified entry at 86.0%. The useful part is not the climb, it is how the maintainers keep score: they publish results they evaluated themselves, under unified settings, separately from results vendors report about themselves. Checking which list a number sits on takes a minute. The benchmark has also been reissued twice in that time, as OSWorld-Verified in July 2025 and then as a separate OSWorld 2.0 in June 2026, so the name alone does not fix what was measured. We wrote about where computer-use agents actually stand at more length; the point here is narrower. A score without its date, its benchmark version and the name of whoever ran it is not evidence anyone can check.

Nothing improper is necessarily happening. Vendors run their own evaluations under their own conditions, with their own prompting, their own retries and their own definition of success, and those choices are legitimate and material. The question for a buyer is simply whether anyone outside the company has reproduced it.

A diagram showing a vendor's reported score passing through checks on what was measured, who measured it and the denominator, before being tested against the buyer's own fifty cases.

Three things to establish, politely and early. Was the evaluation run by the vendor or by a third party. Is the benchmark public, with a leaderboard anyone can check. And if the number is from a public benchmark, does the vendor's entry appear on the public leaderboard at the figure they are quoting.

What "accuracy" is not telling you

One number cannot describe a system that can be wrong in two directions, and almost every useful AI product can.

A fraud check that flags everything catches all the fraud. It is also useless, and its accuracy figure can still look respectable depending on how it is computed. The pair of numbers that matter are how much of the thing you care about the system finds, and how much of what it flags is real. Push one up and the other falls, and the right balance is a business decision rather than a technical one, because it depends on what a miss costs against what a false alarm costs.

Ask the vendor which they tuned for, and then ask whether it is adjustable. A system with a threshold you control is a different purchase from one with the vendor's preference baked in, and the difference matters most in exactly the workflows people buy these products for.

For anything that retrieves before it answers, there is a second pair. Did it find the right source, and did it answer faithfully from what it found? A single accuracy figure merges two failures that need opposite fixes, which is the argument we set out at length in evaluating retrieval systems. A vendor who reports one number for a retrieval product has either not separated them or has chosen not to.

And ask what the system does when it does not know. Abstaining is a feature. A product that returns a confident answer to everything has moved the cost of its uncertainty onto whoever reads the output, which is usually the person who was supposed to be saved time.

The claims that are not about accuracy at all

A product can be accurate and still be the wrong purchase, and the numbers that decide it are rarely on the same slide.

Latency, measured at the right percentile. An average response time hides the tail, and the tail is what users experience on a bad day. Ask for the 95th percentile under the concurrency you expect, not the mean under theirs.

Cost per item at your volume. Pricing that looks reasonable at a demo volume can be the dominant line at production volume, and a vendor whose own cost scales with usage will eventually need that to show up in yours. The arithmetic belongs in the evaluation, on the same spreadsheet as what an AI feature costs you to run.

What happens to your data. Retention, training use, residency and sub-processing are contractual facts, not features, and they can rule out a product with an excellent score. If you already have customer agreements in place, the constraints are the ones we set out in adding AI to a product that already has customers, and they are not negotiable late.

And what evidence they can give an auditor. If you are heading towards a governance standard, the vendor is inside your scope, and the ISO 42001 expectation of documented, repeatable risk assessment extends to what you bought as well as what you built.

The denominator is where the claim hides

Most inflated claims are not false. They are true about a smaller population than the listener assumes.

"95% accurate" on what proportion of inputs? A document system that processes the clean 70% automatically and routes the rest to a human is a genuinely useful product, and its accuracy figure describes the 70%. That is a perfectly good system and a perfectly misleading slide, depending on whether the routing is mentioned.

Accuracy on which classes? A classifier that is 98% accurate on a dataset where 97% of cases are the common one has told you almost nothing. The number that matters is performance on the rare class, which is usually the reason you are buying.

Measured against whom? A system compared against an untrained human performs differently from one compared against your best operator. Both comparisons are real and they answer different questions.

Over what period? A number from a controlled evaluation window is not a number from eight months of drift. Model versions move underneath a product, as a model retirement forces, and a score from last year's version is a historical fact rather than a current one.

What should I ask an AI vendor before buying?

Six questions, and the quality of the answers tells you more than the answers do.

The six questions

  1. What exactly was measured, and on which subset of inputs?
  2. Who ran the evaluation?
  3. What happens to the cases the system cannot handle?
  4. What does a failure cost in your workflow?
  5. How will we measure this in production?
  6. What would make you tell us this is not a fit?

The last one is the most informative. A vendor with a clear answer has thought about where their product fails, which is the thing you most need to know and the thing no benchmark reports. A vendor who says there is no such case is either selling or has not looked.

Ask about the hand-off, not just the accuracy. Every real deployment has a path for the cases the system cannot do. How those are detected, what the confidence signal is, where they go and how long they wait are the operational questions that decide whether the product saves anyone time. A system with slightly lower accuracy and a reliable hand-off beats a more accurate one that fails silently.

And ask what they measure in production, because a vendor who only measures at evaluation time has no way to tell you when the system degrades. That is the same argument we make about tracing and observability for your own builds, and it applies with more force to something you do not control.

An invitation to discuss evaluating an AI vendor against your own cases with the Zyneto team.

Run your own benchmark, because theirs is not yours

This is the step that replaces the argument, and it is less work than the procurement process it short-circuits.

Fifty real cases with known answers. Drawn from your own records, including the awkward ones, the badly formatted ones and the ones your own staff disagreed about. The habit is the same one we recommend for evaluating an agent you build yourself: real traces beat invented cases, because invented cases reflect what you imagined rather than what arrives.

Hold some back. If the vendor tunes against the set you gave them, the result measures tuning rather than capability. A reserved portion they never see is the control, and it costs nothing to keep.

Include the cases you expect to fail. A pilot designed to succeed produces a number that means nothing and a decision nobody can defend later.

Do not generate the cases with a model. Cases produced by a model land inside what models already handle well, which inflates the score and teaches you nothing, as the limits of synthetic data make plain. Generated cases are acceptable as scaffolding when you genuinely have no data; they are not acceptable as the evidence for a purchase.

Count the cases nobody can agree on. When two of your own people disagree about the right answer, that is not a flaw in the test set, it is a measurement of how hard the task actually is, and it sets a ceiling no vendor can beat. A task where your experts agree 85% of the time cannot support a 95% accuracy claim from anyone, and knowing that number before the meeting changes what you are willing to believe. It is also the first thing getting your own data ready surfaces, which is why that work pays for itself before any purchase.

Score it yourself, with the definition agreed in advance. Half of all disagreements about an evaluation are disagreements about what counts as correct, and they are much easier to settle before the result is known.

The pilot that proves nothing

Vendors offer pilots and most pilots are designed to be passed. The pattern is familiar: a hand-picked dataset, vendor engineers on hand, a success criterion agreed afterwards.

A comparison of a pilot designed to be passed against a pilot that can produce a defensible decision.

A pilot worth running has four properties. The data is yours and chosen before the vendor sees it. The success threshold is written down first, with a number and a date. The vendor's involvement is the involvement you will have in production, not a temporary team. And there is a defined failure condition that ends the pilot rather than extending it.

We have made this argument about internal projects, where the pilot that never reaches production is usually one that could not fail by construction, and the reasoning is identical when the pilot is a vendor's. Omdia's 2025 AI Market Maturity Survey, covering 448 enterprises, found that proof-of-concept failures mostly come from enterprises and vendors misjudging what deployment involves rather than from defects in the technology being tested. The number on the slide is rarely the thing that goes wrong.

Is a high benchmark score worth paying for?

Only if the benchmark resembles your work and the score is reproducible by someone other than the seller. A model that leads a public leaderboard on reasoning tasks may be no better than a cheaper one at classifying your support tickets, and you will pay for the leadership either way.

The decision that actually saves money is to measure the cheapest adequate option against your fifty cases first. If it passes, the more expensive system has to justify its margin against a measured baseline rather than against a slide. That is the same discipline that decides what an AI feature costs you to run, and it is routinely skipped because the better score feels like the safer choice.

An invitation to scope an evaluation of an AI vendor against your own data with the Zyneto team.

What we would do

Build the fifty cases before talking to any vendor. They are reusable across every vendor you consider, they outlast the purchase, and they become the regression suite that tells you when the product gets worse. That single artefact is worth more than the procurement exercise around it.

Ask the three questions in the first meeting: what was measured, who measured it, what is the denominator. Then ask the sixth question, about when the product is not a fit, and listen to how long the answer takes.

Treat the pilot as an experiment with a pre-registered hypothesis rather than as a sales stage. Write the threshold and the date before the data moves.

Write down what you decided and why. Six months on, when the product is disappointing somebody, the useful artefact is the record of what you tested, what threshold you set and what you accepted. Without it the conversation becomes a dispute about memory, and the vendor has better notes than you do. A single page naming the test set, the threshold, the result and the known limitations is enough, and it is the page a procurement review or an auditor asks for later.

And keep measuring after you buy. A vendor's accuracy claim describes a moment; your production data describes the thing you are paying for, and the two diverge quietly. The organisations that are happy with their AI purchases two years later are, almost without exception, the ones that never stopped checking.

Frequently asked questions

Establish three things before anything else: what exactly was measured, who ran the measurement, and what the denominator is. A reported score describes a fixed benchmark under the evaluator's own conditions, including their prompting, retries and definition of success. Then stop debating and run the system on fifty of your own real cases with known answers, scored against a definition of correct that both sides agreed before the result was known.

Researchers working on detection propose defining it by effect rather than cause: contamination is artificially inflated, non-generalizing benchmark performance, where a score does not carry over to rephrased samples, synthetic samples from the same distribution, or other benchmarks for the same task. For a buyer that is exactly the right question, because transfer is what you are purchasing. Separate work concludes the effect may be considerably larger than recent model releases report.

Not reliably. A benchmark measures a fixed set of tasks chosen by someone else, and leadership on reasoning tasks says little about classifying your support tickets or reading your documents. The practical move is to measure the cheapest adequate option against your own cases first, so any more expensive system has to justify its margin against a measured baseline rather than against a slide.

Ask whether the evaluation was run by the vendor or a third party, whether the benchmark is public with a leaderboard anyone can check, and whether the vendor's entry appears on that leaderboard at the figure being quoted. Some benchmarks make this easy: the OSWorld desktop-agent project publishes the results its own team evaluated under unified settings separately from the results vendors report about themselves, so you can see which list a quoted number came from.

Four properties. The data is yours and selected before the vendor sees it. The success threshold is written down first, as a number with a date. The vendor's involvement during the pilot matches what you will get in production rather than a temporary team of their engineers. And a failure condition is defined in advance that ends the pilot instead of extending it. A pilot that cannot fail produces a decision nobody can defend later.

Only partly, and hold some back. Vendor input on format and edge cases is useful, but if they can tune against every case you supply, the result measures tuning rather than capability. Reserve a portion they never see as a control. Also avoid generating cases with a model: generated cases cluster inside what models already handle well, which inflates the score and tells you nothing about the inputs that actually arrive.

The same fifty cases, on a schedule, plus production outcomes. A vendor's claim describes one moment under their conditions, while model versions, prompts and your own data all move afterwards. Re-running a fixed set gives you a comparable number over time, and production measurement tells you about the inputs the test set never contained. Organisations still satisfied with an AI purchase after two years are generally the ones that never stopped checking.

Vikas Choudhary

Vikas Choudhary

Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.

Zyneto builds custom software, AI and automation

Tell us what you are building and we will reply with a clear next step.

Phone

We reply to every enquiry within one business day.

WhatsApp
Email
Book a Meeting