AI agents do more than answer questions. They issue refunds, update records, and hand work to other agents, and a transcript that looks perfect can hide a refund that happened twice.
AI Agent Evals is a practical field guide to evaluating agents by what they changed in the systems they touch. Its central idea, the Evidence Ladder, gives product managers and AI engineers one rule they can apply in any design review: an agent should act on its own only up to the highest level of proof you can verify automatically.
The book follows one agent from start to finish: Speedy, the customer support assistant at Pronto, a fictional online grocery store. Every chapter tests one of Pronto's real policies, from the $50 refund rule to substitution opt-ins and safety escalations.
You'll learn how to:
Write task contracts that define success, including correct refusals and escalationsFind what breaks with error analysis, before choosing any metricBuild LLM judges, then grade them against expert labelsGrade tool calls and trajectories without dictating the pathRed-team boundaries, permissions, guardrails, and data leakageEvaluate RAG, memory, handoffs, and multi-agent systemsMonitor production, catch drift, and gate releases in CI/CDChoose between large and small models, and route between them, on evidencePromote an agent's autonomy only as far as the evidence supportsMake the release call, ship, limit, or hold, with a gate table written before the runEvery example is runnable Python, and a companion repository on GitHub runs the whole book end to end.