Article · Architecture & Technology
AI Agent Evaluation: How to Measure If an Agent Does Its Job Well
Evaluating an AI agent means measuring, against test cases with known answers and per-task metrics, whether it does its job well enough for production, and measuring it again every time the model or the process changes. It is not running the happy path in a demo. It is the thing that separates a system that looks like it works from one that does, and the first thing I check before letting an agent touch a real operation.
I put it that bluntly because the market sells demos, and demos always go well: you pick the clean case, run it once, and everyone claps. Real operations are not that. They are case number three hundred, the one that arrives with a mistyped field, an exception nobody documented, and a deadline on top. Without serious evaluation, you have no way of knowing what your agent does with that case until it does it wrong in front of a customer.
What evaluation is, and what it isn’t
Evaluating answers a question a board can sign off on: across a representative set of cases, how often does the agent do the right thing, and what happens when it isn’t sure? Everything else is a nuance of that question.
It is not the same as AI agent observability, which watches the system while it runs live. Evaluation runs against a controlled set of cases with known answers, mostly before the agent sees real work; observability watches what happens afterward, with real traffic. Both are necessary and they feed each other: the failures observability catches become new cases in the evaluation set.
The evaluation set: cases with known answers
The foundation is a set of test cases where you already know the correct answer. For an agent that reconciles invoices, that means a batch of real, anonymized invoices with the outcome a good accountant would give each one: this one matches, this one has a freight discrepancy, this one has to be escalated because the order isn’t there. The agent processes the batch and you compare its output to the expected answer, case by case.
That set is built deliberately, not scraped together. It has to include:
- Normal cases, the day-to-day ones, which should come out right almost always.
- Known exceptions, the ones the process already knows how to handle and the agent must recognize.
- Edge and ambiguous cases, where the correct answer is often “this isn’t mine to decide, I escalate it.”
- Trap cases, malformed or incomplete inputs that show up in production more than anyone admits.
A set with only easy cases lies: it returns a high score and says nothing about real risk. The quality of the evaluation depends on the hard cases being in there. And that set comes from your operation, not from a generic catalog: it is part of the operating architecture codified when the process is mapped, an asset that belongs to the client, not to the platform.
Per-task metrics, not one global score
“The agent is 90% accurate” means nothing until you say at what. 90% accuracy with 10% silent errors reaching the customer is unacceptable. 90% with the other 10% escalated to a person can be excellent. The useful metric is defined per task and by what each type of error costs.
| Metric | What it measures | Why it matters |
|---|---|---|
| Accuracy by case type | How often the output matches the correct answer, broken down by category | A global average hides that it fails exactly on the expensive cases |
| Exception coverage | What share of known exceptions it recognizes and handles well | Measures whether the agent survives real operations, which live on exceptions |
| Correct escalation rate | How often it escalates when it should, and how often it over-escalates | An agent that never escalates is dangerous; one that escalates everything is useless |
| Impactful false positives | Wrong actions that reach the system or the customer unfiltered | This is the error a board won’t forgive; it gets watched separately |
The distinction between an error that stays inside and one that gets out is what governs risk. That’s why I always separate escalation. I prefer an agent that stops when in doubt and hands the case to a person, with the context prepared, over one that decides for the sake of deciding. Where that line sits is a business decision, and I develop it in human in the loop; the mechanics of who gets escalated to and how, in exception handling and human escalation.
Before production and continuously
Evaluation happens at two moments, and both are mandatory. Before it gets real work, the agent runs the full set and has to clear agreed thresholds. Below that quality it does not go into production, full stop. That gate is what turns a proof of concept into a system you can answer for in front of a board. Many pilots die precisely here, because there never was a threshold. The demo was enough to excite people and nobody measured whether it held up at volume. I’ve written about that pattern in why enterprise AI pilots fail.
The second moment never ends. An agent in production is not a machine you install and forget: the model underneath gets updated, your process changes, a type of case appears that didn’t exist before. Each of those changes can degrade performance without anyone noticing, and the only way to know is to run the evaluation again. In Arkatai that is part of the operate-and-update cycle: when a piece changes, the evaluations are rerun before anyone trusts the result.
What happens when you change the model
This point deserves its own paragraph because it’s where most people get complacent. Swapping the model under an agent, because a better, cheaper, or faster one came out, looks like a minor technical change. It isn’t. A different model can do better on normal cases and, at the same time, handle one specific exception worse than the previous one nailed. Without evaluation, that regression surfaces in production, on a real case.
That’s why I treat models as an interchangeable utility, but I never swap one without rerunning the full evaluation set. The ability to change models without redoing the process is an advantage. Exercising it without measuring is recklessness. The criteria for choosing a model per task I develop in the AI agent architecture hub, where evaluation fits inside the whole system.
Evaluation, observability and auditing: three different things
These get confused, so I’ll separate them. Evaluation measures quality against known cases, mostly before production. Observability watches the system while it runs. Auditing reconstructs after the fact what the agent decided and why, as evidence for a third party. I cover it in auditing AI agent decisions, and the control framework that wraps all of them in AI agent governance. All three rest on the same trace, but they answer different questions: is it good, is it going well right now, can I prove what it did? A serious system needs all three.
If I had to leave a board with one idea, it’s this: don’t ask whether the agent is good, ask to see how good is measured, on which cases, and against what threshold. The answer to that question separates whoever has a system from whoever has a lucky demo. This article belongs to the AI agents for business cluster, where the rest of the adoption decisions fit.
Frequently Asked Questions
What is AI agent evaluation?
It is measuring, against a set of test cases whose correct answer you already know, whether the agent does its task well enough to operate. It’s done before giving it real work, to decide whether it enters production, and continuously afterward, every time the model or the process changes.
What metrics are used to evaluate an agent?
Not one global score, but per-task metrics: accuracy broken down by case type, coverage of known exceptions, correct escalation rate, and impactful false positives. The key is separating errors that stay inside the system from those that reach the customer, because they cost very different things.
Why re-evaluate when you change the model?
Because a new model can improve on normal cases and, at the same time, handle an exception the previous one solved well worse. That regression is invisible without rerunning the full evaluation set. Finding it in production means finding it on a real case with a customer in front of you.
How is evaluation different from observability?
Evaluation measures quality against cases with known answers, mostly before production. Observability watches the system while it runs live, with real traffic. They complement each other: the failures observability catches become new cases in the evaluation set.