How to Evaluate and Test Your AI Agents: A Practical Guide to Evals

You built an agent. It calls tools, chains reasoning steps together, and produces answers that look right most of the time. You show a demo. Everyone's impressed.

Then it goes to production and starts hallucinating account balances, calling the wrong API endpoints, and confidently giving users the exact opposite of what they asked for. Nobody catches it for three days.

This is the reality for most teams shipping AI agents today. Traditional software testing assumes deterministic outputs — the same input always produces the same output. Agents break that assumption entirely. And most teams respond by... not testing them at all.

What Makes Agent Evaluation Different

Testing a REST API is straightforward. You send a request, you check the response, you move on. Agents are a different beast for several reasons:

• Non-deterministic outputs — the same prompt can produce different responses across runs, making exact-match assertions useless for most cases
• Multi-step reasoning — an agent might take five steps to answer a question, and a failure at step two silently corrupts everything downstream
• Tool use — agents don't just generate text; they call functions, query databases, and hit APIs, which means failures can happen at the boundary between the LLM and external systems
• Compounding errors — unlike a single LLM call, agentic loops amplify small mistakes with each iteration

A wrong word in a chatbot response is annoying. A wrong tool call in an agent that manages your calendar, sends emails, or queries your database is a real problem.

The Three Levels of Agent Evals

Not all evals test the same thing. A useful mental model breaks agent evaluation into three distinct levels, each catching different types of failures.

Unit-Level Evals

Test individual components in isolation. Does the prompt produce the right format? Does the retrieval step return relevant documents? Does the agent select the correct tool for a given query?

These are the closest to traditional unit tests and the easiest to write. If your agent uses a RAG pipeline, you can test retrieval precision independently from generation quality.

Trajectory-Level Evals

Test the path the agent takes, not just the destination. Did it call tools in a reasonable order? Did it ask for clarification when the input was ambiguous instead of guessing? Did it avoid unnecessary steps?

Trajectory evals are critical for multi-agent systems where one agent delegates to another. The final answer might be correct, but if the agent took 47 steps and $3 in API calls to get there, you have a problem.

End-to-End Evals

Treat the entire system as a black box. Give it a task, check the output. Did the agent accomplish what the user wanted?

This is the eval that matters most to users, but it's also the hardest to automate. A task like "book me a flight to Berlin next Tuesday" has a clear success condition. A task like "summarize this research paper" requires judgment.

Start with end-to-end evals for coverage, then add unit and trajectory evals where failures cluster. You don't need all three from day one.

Evaluation Methods: Picking the Right Judge

Once you know what to evaluate, the question becomes how. There are three main approaches, and most teams need a mix.

Code-Based Assertions

The simplest method. Check for exact matches, regex patterns, JSON schema conformance, or whether specific tools were called. Works well for structured, deterministic outputs — did the agent return valid JSON? Did it call the search_flights function and not the delete_account function?

LLM-as-a-Judge

Use a separate LLM to score open-ended outputs. Techniques like G-Eval ask a model to rate responses on criteria like relevance, correctness, and helpfulness. You can also run pairwise comparisons — show the judge two outputs and ask which is better.

This method scales well and handles the nuance that code-based checks can't. The tradeoff: your eval is now only as good as your judge model. OpenAI's eval best practices recommend using a stronger model as judge than the one being evaluated.

Human Review

For high-stakes decisions — financial advice, medical information, safety-critical actions — humans need to be in the loop. Not for every test case, but for a sampled subset that keeps your automated evals honest.

The smart move is to use human review to calibrate your LLM-as-judge scores. If the judge says an answer is 9/10 but human reviewers consistently rate it 6/10, you know your automated eval is miscalibrated.

Method	Best For	Scales?	Cost
Code-based assertions	Structured outputs, tool calls, format checks	Yes	Low
LLM-as-a-judge	Open-ended quality, relevance, reasoning	Yes	Medium
Human review	High-stakes, calibration, edge cases	No	High

Tools That Make Evals Practical

You don't need to build an eval framework from scratch. Several mature tools exist, ranging from open-source CLIs to full platforms.

• Braintrust — end-to-end eval platform with a native GitHub Action for CI/CD. Posts score comparisons directly on PRs. Its AI assistant can auto-generate test datasets and optimize prompts. Best for teams wanting a managed, full-stack solution.

• Promptfoo — open-source CLI that runs entirely locally. Declarative YAML configs define test cases and assertions. Produces side-by-side output matrices across models. Includes built-in red-teaming and security testing that other tools lack. Best for developers who want full control.

• DeepEval — open-source Python framework that works like pytest for LLMs. Ships with 30+ built-in metrics covering hallucination, relevancy, faithfulness, and tool correctness. Best for Python-heavy teams who want to integrate evals into existing test suites.

• LangSmith — tracing and evaluation platform built for LangChain and LangGraph. Combines detailed agent traces with dataset management and automated scoring. Best if you're already in the LangChain ecosystem.

• OpenAI Evals — OpenAI's native evaluation toolkit. Tightly integrated with their API and models. Best for teams exclusively using OpenAI models.

Tool	Open Source	CI/CD Integration	Key Strength
Braintrust	No	Native GitHub Action	Full platform + prompt optimization
Promptfoo	Yes	GitHub Action + CLI	Red teaming + local-first
DeepEval	Yes	Pytest integration	30+ research-backed metrics
LangSmith	No	LangChain native	Tracing + debugging
OpenAI Evals	Partial	API-based	OpenAI ecosystem integration

Building Your Own Eval Strategy (Without Overengineering It)

Most teams fail at evals not because they lack tools, but because they try to build a comprehensive evaluation suite before they have a single test case. Here's a practical starting point:

1. Write 5-10 golden test cases by hand. Real queries your agent will encounter, paired with what a good response looks like. This is your baseline.
2. Add code-based assertions for anything deterministic. Tool selection, output format, required fields — these should never regress.
3. Layer in LLM-as-a-judge for open-ended quality. Pick one or two metrics that matter most for your use case. Not ten. One or two.
4. Run evals in CI. Every PR that touches your agent's prompts, tools, or logic should trigger an eval run. Both Braintrust and Promptfoo make this straightforward.
5. Review failures weekly. Your golden test cases will drift as your agent evolves. Update them. Remove ones that no longer reflect real usage. Add new ones from production failures.

The goal isn't perfection. It's catching regressions before they reach users and building confidence that changes improve things rather than break them.

If you're building agents from scratch, bake evals into the process from the start. Retrofitting them later is always harder.

What Happens When You Skip Evals

The failure modes are predictable and painful:

• Silent regressions — a prompt tweak that improves one case quietly breaks five others, and nobody notices for weeks
• Hallucination drift — the agent's accuracy slowly degrades as the underlying model gets updated, with no baseline to compare against
• Broken tool calls — the agent starts passing malformed parameters to external APIs, causing data corruption that's expensive to unwind
• Guardrail failures — safety boundaries that worked in testing silently stop triggering in production edge cases

Every team that ships agents to production eventually builds an eval pipeline. The only question is whether you build it before or after the first incident. Build it before.