How to Write Evals for Your AI Agent

Almost everyone building AI agents skips the same step, and it's the one that separates a demo from a product: evaluation. They build the agent, try it on a few examples, watch it work, and ship. Then it fails in production in a way they never tested for, and they have no systematic way to know whether the fix they made helped or quietly broke three other things.

Evals are how you escape that trap. They're the difference between it seemed to work when I tried it and it works, and I can prove it, and I'll know immediately if a change makes it worse. Here's how to actually build them for an agent.

Why agents are harder to evaluate than single calls

Evaluating a single model call is relatively easy: you have an input and an expected output, and you check whether they match. Agents break that model in two ways.

First, there's no single right answer. An agent pursuing a goal might take several valid paths to get there. Exact-match checking is useless when the correct output is a range of acceptable ones.

Second — and this is the crucial one — an agent can produce the right answer for the wrong reasons, or a wrong answer after doing everything right. It might call the correct tools in the correct order and still botch the final response. Or it might luck into a passable answer while its reasoning was nonsense. So you have to evaluate not just the destination but the journey: the reasoning, the tool calls, and the final result. That trajectory is what makes agent evaluation its own discipline.

The three things worth measuring

1. Task completion — did it achieve the goal?
The headline metric. Infer what the user actually wanted, then judge whether the agent's final result met that goal. Note this is about the outcome, not string-matching an expected output — a support agent that resolves the issue in wording you didn't predict still succeeded.

2. Tool use — did it use the right tools, correctly?
Agents act through tools, so tool behavior is a first-class signal. Did it choose the right tool for the step? Pass well-formed arguments? Avoid calling tools it didn't need? A surprising share of agent failures are tool failures — wrong tool, malformed call — and these are often cheap to check mechanically. (Enforcing argument shape with strict schemas prevents a whole category of them.)

3. Reasoning / trajectory — did the path make sense?
Evaluate the thinking between steps: was each decision relevant to the goal, and did the chain stay coherent across the run? This is what catches the right answer, wrong reasons case that will eventually bite you on an input you didn't test.

The two tools you'll use to grade

Deterministic checks — run these on 100% of outputs. They're code, so they cost essentially nothing: did the output parse as valid JSON? Did the tool calls have the right shape? Is the output within length bounds? Did it pass safety filters? These catch the most common, most embarrassing failures for free. Always start here — every eval suite should have a layer of cheap, mechanical checks.

LLM-as-judge — for the things code can't measure. Task completion and reasoning quality are semantic and subjective; you can't regex them. So you use another model as a grader: give it the agent's input, its trajectory, and its output, plus a clear rubric, and have it score the result. The keys to making this reliable:

• Write an explicit rubric with score thresholds. Vague rate this 1–10 produces noise. 0.8–1.0 = fully resolved the user's issue; 0.5–0.7 = partially; below 0.5 = failed produces consistent, reproducible scores.
• Ask for a short explanation with the score. It makes the judge's reasoning auditable and helps you spot when the judge is wrong.
• Use a capable model as the judge for hard semantic calls — but you can route cheaper, high-volume checks to a smaller judge model to keep costs sane.

The smart pattern is a funnel: cheap deterministic checks on everything, a lightweight judge for common semantic checks at scale, and your strongest judge reserved for the genuinely hard calls. Spend compute where it produces the most signal.

How to actually get started

You don't need a fancy framework to begin. You need:

1. A dataset of realistic tasks — 20–50 inputs that look like what your agent will really face, including the messy edge cases you're worried about. This is the highest-leverage thing you'll build; a small set of representative cases beats a big set of easy ones.
2. Deterministic checks for the mechanical stuff — write them first, they're cheap and catch a lot.
3. An LLM judge with a written rubric for task completion and reasoning quality.
4. A number you track over time. Run the suite, get a score, and re-run it on every meaningful change. The point isn't a perfect grade — it's movement: did this change make the agent better or worse?

That's it. Even this minimal setup transforms how you build: every change becomes measurable instead of vibes-based.

The takeaway

Evals are the unglamorous discipline that makes agents trustworthy. They're harder than single-call evaluation because you have to judge the whole trajectory — task completion, tool use, and reasoning — not just a final string. But the payoff is enormous: you stop guessing whether your agent works and start knowing, and you catch regressions the moment they appear instead of when a user reports them. Build the smallest version now — a handful of real tasks, some cheap checks, one judge with a rubric — and grow it as you go.

This is the reliability half of building agents. For the build itself, start with How to Build Your First AI Agent, and before you commit to an agent at all, make sure you need one: AI Agent vs Workflow vs Single Call. Prompts to build and test with are in the prompt library.