How to tell whether an agent is any good

A good demo is not an evaluation

An agent that worked well three times in front of you has shown you very little. Its steps vary from run to run, a small prompt change can shift behavior in places you did not test, and it will cheerfully tell you it is finished when it is not. An eval is a set of tasks with a way to score each result automatically, so that you can change something and learn whether it got better.

Everything in this lesson comes from Anthropic's guide "Demystifying evals for AI agents" and the evals described inside their own agent projects.

Three kinds of grader

A grader looks at what the agent did and produces a score. There are three types, and each pays for something different.

• Code-based graders check a fact: did the test pass, does the file exist. They are cheap and fast, and brittle. A correct answer worded in an unexpected way can fail.
• Model-based graders use another model with a rubric. They are flexible and can read open-ended output. They need calibration against human judgment or they drift.
• Human graders are the gold standard and the slowest.

The research system uses a model judge scoring from 0 to 1 on accuracy, citations, completeness, source quality and tool efficiency. It still needed people: humans caught a bias toward SEO-farm pages.

Grade the outcome, not the path

Agents find routes you did not expect. If your grader insists on a specific sequence of tool calls, a better route fails the test. The guide's rule is to grade the result in the world: is the reservation in the database, did the file change, does the program build.

The compiler project shows how far this can go. The agents were given GCC as a known-good reference. Because the check was nearly perfect, agents could isolate kernel bugs in parallel. Carlini's warning follows from that: the agent will solve whatever the check measures, so a weak check produces confident, wrong work.

Where tasks come from, and how many

Start from real failures. Take the cases where the agent actually let someone down, and make each one a task. The guide suggests 20 to 50 such tasks. Anthropic's research team also began with about 20 queries.

Since agents are not deterministic, you run each task several times, and the numbers behave in a surprising way. With pass@k, you count success if any of k tries works, so the score rises as k grows. With pass^k, every try must work, so it falls. An agent that succeeds 75% of the time per trial passes three in a row only about 42% of the time. Which one you care about depends on whether a user can retry.

The evaluator also needs an eval

A separate evaluator agent is a strong tool, but it starts out lenient. In the harness project, theirs approved bugs it had itself found. It took several rounds of reading its verdicts and adjusting its prompt, against what a person would decide, before it was skeptical enough to use.