Study notes · three weeks · 3–4 h a week
Agent Evaluation
Agents fail in the middle: a wrong tool, a bad argument, a loop. This path teaches evals that look at the whole run, then puts them on a platform and in CI so they gate releases.
The core material
| Resource | Role | Use it for |
|---|---|---|
| Demystifying evals for AI agents | Mental model | Tasks, trials, graders, pass@k vs pass^k, capability vs regression suites (Anthropic, Jan 2026) |
| Evaluating AI Agents | Guided hands-on | About 2.5 h of notebooks with Arize: tracing, router and skill evals, trajectory evals, judge tuning |
| Azure AI Foundry agent evaluators | Enterprise platform | Built-in task, intent and tool-use evaluators you can run on Azure |
| LLM Evaluation (Educative) | Subscription course | About 2 h on error analysis, pass/fail grading, agent workflows and evals in CI |
- Read the transcripts. Every resource here says the same thing in different words: a score without the trace behind it hides whether the agent or the grader is wrong.
01 / Week 1
How agent evals differ
Learn the vocabulary and the shape of a good eval suite before writing a single grader.
| Concept | What it means | Why it matters |
|---|---|---|
| Task | One test case: inputs plus success criteria | The unit you count and track |
| Trial | One attempt at a task | Agents vary run to run, so run several |
| pass@k | At least one of k trials succeeds | Fine when one success is enough |
| pass^k | All k trials succeed | The bar for customer-facing agents |
| Capability eval | What the agent can do; starts low | Shows where to improve |
| Regression eval | What it must keep doing; near 100% | Gates every release |
Read (~1.5 h)
- Demystifying evals for AI agents: tasks and trials, code vs model vs human graders, why you read transcripts
- Agent trajectory evaluations (Arize): scoring the sequence of steps, not only the result
Do (~2 h)
- Evaluating AI Agents, lessons 1–6: decompose an agent, build it, then trace every step
- LLM Evaluation (Educative): the error-analysis and pass/fail modules
CheckpointExplain when you would report pass^k instead of pass@k, with an example from an agent you have built.
02 / Week 2
Tool-call, trajectory and judge evals
Grade the parts an agent can get wrong: routing, tool choice, arguments, use of results, and the path as a whole.
Do (~3 h)
- Evaluating AI Agents, lessons 7–13: router and skill evals, trajectory evals, structured experiments, improving the LLM judge
- Write three graders for one of your agents: a code check on tool arguments, a trajectory check against an expected tool sequence, and an LLM judge for the final answer.
- Hand-label 20 runs and measure how often your judge agrees with you before trusting it.
Rule of thumb: Use code graders wherever the answer is checkable; save model graders for open-ended output, and calibrate them against human labels.
CheckpointYour judge agrees with your hand labels on at least 80% of runs, and you can show one failure it caught that a final-answer check missed.
03 / Week 3
Evals on Azure AI Foundry and in CI
Run built-in agent evaluators on Azure, then make a regression suite block a bad release.
| Foundry evaluator | What it checks |
|---|---|
| Task Completion | Did the agent deliver what the user asked for |
| Task Adherence | Did its actions follow the system message and policy |
| Intent Resolution | Did it identify and address the real intent |
| Tool Selection | Right tools, no unnecessary ones |
| Tool Input Accuracy | Correct parameters on every call |
| Tool Output Utilization | Did it use tool results correctly |
| Task Navigation Efficiency | Was the tool sequence efficient against ground truth |
Source: Foundry agent evaluators
Do (~3 h)
- Agent evaluators reference: several are marked preview; check status before relying on one
- Evaluating AI Agents, lesson 14: monitoring agents after launch
- Flow Framework testing: scenario files where a model plays the user, another plays the tools, and a judge grades the run
- LLMOps: Building Production-Ready LLM Systems (Educative): golden datasets and LLM-as-judge in a production pipeline
CheckpointA CI job runs your regression suite on every pull request and fails the build when pass^3 drops.
Capstone
Ship an eval suite for one real agent you own. Done when every line holds:
- At least 30 tasks, split into a capability set and a regression set
- Each task runs at least 3 trials, and I report pass@k and pass^k
- Code graders for every checkable step; a calibrated judge for the rest
- At least one trajectory check against an expected tool sequence
- The regression set runs in CI and blocks merges
- I have read 20 failing transcripts and fixed either the agent or the grader