# How to tell whether an agent is any good

Judge an agent by what it leaves behind, not by the steps it took. Start from real failures, mix cheap graders with calibrated ones, and suspect the test before the agent.

## A good demo is not an evaluation

An agent that worked well three times in front of you has shown you very little. Its steps vary from run to run, a small prompt change can shift behavior in places you did not test, and it will cheerfully tell you it is finished when it is not. An **eval** is a set of tasks with a way to score each result automatically, so that you can change something and learn whether it got better.

Everything in this lesson comes from Anthropic's guide "Demystifying evals for AI agents" and the evals described inside their own agent projects.

## Three kinds of grader

A **grader** looks at what the agent did and produces a score. There are three types, and each pays for something different.

- **Code-based graders** check a fact: did the test pass, does the file exist. They are cheap and fast, and brittle. A correct answer worded in an unexpected way can fail.
- **Model-based graders** use another model with a rubric. They are flexible and can read open-ended output. They need calibration against human judgment or they drift.
- **Human graders** are the gold standard and the slowest.

The research system uses a model judge scoring from 0 to 1 on accuracy, citations, completeness, source quality and tool efficiency. It still needed people: humans caught a bias toward SEO-farm pages.

## Grade the outcome, not the path

Agents find routes you did not expect. If your grader insists on a specific sequence of tool calls, a better route fails the test. The guide's rule is to grade the result in the world: is the reservation in the database, did the file change, does the program build.

The compiler project shows how far this can go. The agents were given GCC as a known-good reference. Because the check was nearly perfect, agents could isolate kernel bugs in parallel. Carlini's warning follows from that: the agent will solve whatever the check measures, so a weak check produces confident, wrong work.

```flow
Pick a real failure
Turn it into a task with a checkable outcome
Run the agent several times
Score the end state with a grader
Change one thing and run again
```

**Question:** An agent reaches the right database state by a route your test did not expect. How should the grader behave?

- Fail it, because the steps differed
- Pass it, because the outcome is what matters
- Ask a human every time
- Ignore the result

**Answer:** Pass it, because the outcome is what matters. The guidance is to grade outcomes, such as whether the reservation exists, not the exact path. Requiring a fixed path punishes valid solutions.


## Where tasks come from, and how many

Start from real failures. Take the cases where the agent actually let someone down, and make each one a task. The guide suggests 20 to 50 such tasks. Anthropic's research team also began with about 20 queries.

Since agents are not deterministic, you run each task several times, and the numbers behave in a surprising way. With **pass@k**, you count success if any of k tries works, so the score rises as k grows. With **pass^k**, every try must work, so it falls. An agent that succeeds 75% of the time per trial passes three in a row only about 42% of the time. Which one you care about depends on whether a user can retry.

> **Zero percent is a clue about the test**
If an agent fails every trial across many runs, the task is usually broken, not the agent. In one case, Opus 4.5 went from 42% to 95% on the CORE-Bench benchmark after bugs in the grader and the setup were fixed.


## The evaluator also needs an eval

A separate evaluator agent is a strong tool, but it starts out lenient. In the harness project, theirs approved bugs it had itself found. It took several rounds of reading its verdicts and adjusting its prompt, against what a person would decide, before it was skeptical enough to use.

> **Calibrate the judge against yourself**
Before you trust a model-based grader, compare its verdicts with your own on a handful of runs. Adjust its prompt until they agree. That is the step the harness team repeated before their evaluator was usable.


> **Make your first task**
Think of one time an agent got a job wrong for you. Write the task in one sentence, and then write the check a program could run to see the correct result. If you cannot write that check, that is the thing to solve first.


**Question:** An agent scores 0% across many trials on a task. What does the guide suggest checking first?

- Whether the task or grader is broken
- Whether the model is too small
- Whether you used enough tools
- Nothing, 0% is a valid result

**Answer:** Whether the task or grader is broken. A 0% pass rate over many trials usually means a broken task, not a weak agent. Benchmark score jumps came from fixing graders and setup.