Skip to Content

Evaluating agents: measure what goes wrong, don't guess

Pull failures from the logs, turn them into tests and retest after each change. That is how agents improve.
November 17, 2026

An AI agent that picks the wrong tool or skips a step is not fixed by tweaking the prompt and hoping. You need to measure what goes wrong and retest after every change. Without a fixed test set, every adjustment remains guesswork.

Why ordinary tests fall short here

An ordinary software test confirms that your code works. For an agent that is not enough: it makes its own decisions, and a failure is often not immediately visible. The answer looks fine, but along the way it took the wrong route.

On 4 November DeepLearning.AI announced a free course on exactly this problem: Evaluating and Improving Agentic Applications, built with Arize AI. The instructor is Laurie Voss, Head of Developer Relations at Arize and co-founder of npm. The approach is useful even if you never take the course.

The approach in four steps

Read the traces

A trace is the complete log of a single run: which model calls, which tools, which intermediate results. In it you spot recurring failures, such as a wrong tool, a missing step or a made-up fact.

Turn them into a test set

Every failure you find becomes a test case that you rerun after every change. That way you see whether a change solves the problem and whether nothing else breaks.

Let code check what code can

Whether a required step was carried out, you test with a simple code check. It is cheap and always gives the same answer.

Use a language model as a judge, but calibrate it first

To judge text, you can use a language model as an assessor, a so-called LLM judge. First give it a handful of examples you have already judged yourself. Only when it agrees with your judgement can you rely on it.

We think: an agent only belongs in daily work once it is settled how you will track down its failures. Evaluation is part of the work, not something that comes after.

What you do now

  • Collect ten to twenty failed runs of an agent you already use, and write down in one sentence per run what went wrong.
  • Agree who manages the test set and that no change goes live until the set has been rerun.
  • Ask your supplier or IT partner how they measure the agent's failures. If you get no concrete answer, you know where to start.

How it works under the hood

From the traces you derive failure categories. For each category you choose: a fixed code check where the answer is unambiguous, a judge with a short rubric and a pass or fail verdict where text is involved. Run the whole set as a regression test: the same cases, after every change, with the result next to the previous one.

This news touches chapter 22 of the course AI(l) or nothing by Newway. If you follow the course, this subject returns there at a calmer pace.

What you can do with this

Want to know what this means for your business? Plan an introduction: half an hour, no sales pitch, but an honest answer to the question where to begin. Rather read on first? The course AI(l) or nothing is free and starts with the basics.

Written in response to a publication by DeepLearning.AI (YouTube): https://www.youtube.com/watch?v=5__Fm2TnPcU