EVALUATING SYSTEMS THAT ACT
AI agents don’t just answer.They act.
Evals show whether those actions work, what they change, and whether they keep working.
Learn how agent evals workA demo shows possibility. An eval measures a distribution.
A polished run can succeed while the same task fails on retry, changes the wrong state, exceeds its budget, or violates policy along the way.
An account-update agent can report success after changing the address, yet clear a consent flag or skip confirmation. One successful response tells you what happened once—not how the system behaves across representative situations and repeated trials.
Evaluate the model together with its prompts, memory, agent software (often called the scaffold), tools, permissions, budgets, environment, and graders. Change one and you change the system under test.
An eval is a repeatable experiment on a configured system acting in a world.
Start with the decision the evidence must support: ship, block, restrict, or investigate. Then give the system representative work, let it act through the tools and constraints it will use, record what it did and what changed, grade the evidence, and repeat.
A response is an artifact. An agent leaves a trail.
This is a difference in typical emphasis, not a hard boundary: output-focused evaluations can also be multi-turn, tool-using, and stateful.
| Dimension | Output-focused eval | Agent-system eval |
|---|---|---|
| Unit | A response | A configured system |
| Context | Prompt and inputs | State, tools, permissions, users |
| Time | Often one turn | A trajectory of dependent steps |
| Evidence | Output match or rubric | Outcome, trace, side effects, cost |
| Variance | Sample-level | Trial and environment-level |
| Failure | Wrong answer | Wrong action, state, path, or stop |
Scroll the table horizontally on narrow screens →
An agent eval is a small world with a witness and a gate.
Across the sources reviewed for this guide, four recurring design questions emerge. Use them as a checklist, not a universal taxonomy.
World
Representative state, tools, data, permissions, users, and threats.
Work
Real goals, known failures, edge cases, and situations where the agent must not act.
Witness
State checks, traces, calibrated judges, and people who can establish what happened.
Gate
The evidence threshold for the decision: ship, block, restrict, mitigate, or investigate.
A stateful test world
Give the agent a stateful world to act in.
A useful eval represents the consequential task, tools, permissions, and starting state—not just the prompt.
Permission model
- Initial state
- Known and resettable
- Outcome
- Verified in the world
- Trace
- Observable actions logged
The final answer is not the final state.
An agent can sound finished while the requested change is missing—or while an earlier action crossed a boundary.
“Done — the account was updated successfully.”CLAIMED SUCCESS
Prefer executable evidence
Use tests, database checks, files, invariants, and tool arguments wherever the result can be verified directly.
Allow many valid paths
Grade the required final state and essential milestones instead of demanding one reference sequence.
Name prohibited actions
Keep prohibited actions and irreversible side effects visible even when the final state looks correct.
One green run is not a guarantee.
Agent behavior can vary across runs, and small errors compound across dependent steps. Repeat important tasks from isolated initial states.
Choose the metric to match the workflow. Usepass@k only when multiple attempts are allowed and a winner can be selected. Use passk when every attempt must work.
Reliability is plural
One pass rate. Two very different claims.
This illustrative calculation holds the per-run pass rate at 80%. At least-one success rises with more attempts; every-attempt success falls.
pass@5: useful when a workflow can select a successful attempt
pass5: useful when every attempt must work
Assumes independent attempts with a constant 80% pass rate. Correlated failures will change both curves.
Across 5 trials, an agent with an 80% per-run pass rate has a 100.0% chance of succeeding at least once, but only a 33% chance of succeeding every time.
“Won’t” and “can’t” are different.
Safe-looking behavior can mean refusal, an effective control, missing tools, poor capability, or a grader that failed to see the harm.
Illustrative comparison
Compare legitimate success and unauthorized outcomes.
Report whether an agent completes legitimate work and whether it creates outcomes it was never authorized to create. The values below are examples, not benchmark results.
Preferred direction: higher legitimate success, fewer unauthorized outcomes.
- Baseline86% legitimate success18% unauthorized outcomes
- Detector82% legitimate success7% unauthorized outcomes
- Least privilege78% legitimate success2% unauthorized outcomes
Read useful work and harmful outcomes separately.
- Legitimate task success
- Harmful-request refusal
- Unauthorized outcomes
- Unauthorized outcomes when capable
- Defense false positives
- Cost and latency
Security controls can reduce attacks while also blocking legitimate work. Least privilege, guardrails, prompts, and tool filtering are part of the tested system and should be reported with the result.
The grader must earn trust.
Benchmarks can fail too: tasks may be impossible, scoring may reject valid solutions, simulators may reveal hidden information, and infrastructure may break.
Grading architecture
Keep the evidence separate.
A single score rarely explains an agent run. Preserve each grader’s finding so teams can see what passed, what failed, and why.
- 01Deterministic
State check
Did the requested world state actually change?
account_status = resolved - 02Policy
Trace constraint
Did the agent stay inside the allowed path?
no restricted tool calls - 03Qualitative
Calibrated judge
Was the resolution correct, clear, and grounded?
rubric evidence attached - 04Escalation
Human review
Can an expert resolve uncertainty or high stakes?
reviewer disposition
Four findings · no hidden aggregate
The evidence record retains four independent findings: a deterministic check of the resulting state, a policy check of the action trace, a calibrated qualitative judgment, and a human review disposition.
- 01
Demonstrate a trusted solution.
- 02
Prove known-bad solutions fail.
- 03
Attack the grader for loopholes.
- 04
Inspect false passes and false failures.
- 05
Compare model-based grading with expert human judgments.
- 06
Version tasks, worlds, graders, and metrics.
Evals are infrastructure, not an event.
Stable suites can catch regressions before release. Production failures and drift should become new test cases.
The operating cycle
A continuous evaluation cycle.
Design defines the work and evidence. Testing runs and inspects repeated trials. Operations uses the results to gate releases, monitor behavior, and refresh the suite.
- 01Map
Decisions, tools, and consequences
- 02Observe
Real tasks and failure traces
- 03Specify
State, constraints, and success
- 04Run
Repeated trials in a resettable world
- 05Grade
Outcome, process, safety, and cost
- 06Inspect
Read traces and challenge graders
- 07Gate
Block regressions before release
- 08Monitor
Watch production drift and incidents
- 09Refresh
Turn new failures into new cases
Twenty-five primary sources behind this guide.
Guidance, benchmarks, datasets, research papers, and evaluation frameworks reviewed on July 24, 2026.
Guidance & risk
Frameworks for designing, calibrating, governing, and operating evaluation programs.
- 01Demystifying evals for AI agentsAnthropic · 2026Defines agent-eval terminology and shows how to combine graders, repeated trials, capability suites, and regression suites.
- 02Evaluate agent workflowsOpenAI · Living docsConnects trace inspection and grading to reusable datasets and repeatable workflow-level evaluation.
- 03Evaluation best practicesOpenAI · Living docsExplains application-specific criteria, representative datasets, human calibration, and continuous evaluation.
- 04AI Risk Management Framework 1.0NIST · 2023Frames evaluation as part of governing, mapping, measuring, and managing multidimensional AI risk.
- 05Generative Artificial Intelligence ProfileNIST · 2024Shows why static benchmarks need deployment-context testing, lifecycle monitoring, and external feedback.
- 06Inspect AIUK AI Security Institute · 2024Provides a concrete evaluation stack spanning datasets, agents, tools, sandboxes, scorers, and auditable logs.
Environments & work
Benchmarks that place agents in reproducible software, browser, desktop, and terminal worlds.
- 01SWE-benchSWE-bench · 2024Tests issue-to-patch software work against fixed repository states and executable outcome graders.
- 02WebArenaWebArena · 2024Introduces reproducible, stateful web tasks graded through database, API, DOM, URL, and answer checks.
- 03OSWorldOSWorld · 2024Evaluates general computer use across real desktop applications with multiple observation modes and execution-based grading.
- 04WorkArenaServiceNow Research · 2024Uses parameterized enterprise workflows and validators to measure agents in complex application interfaces.
- 05WorkArena++ServiceNow Research · 2024Extends enterprise evaluation to compositional knowledge work, implicit instructions, held-out seeds, and task isolation.
- 06BrowserGymServiceNow Research · 2025Standardizes browser observations, actions, manifests, traces, and replay across heterogeneous web benchmarks.
- 07Terminal-Bench 2.0Terminal-Bench · 2026Pairs expert-authored terminal tasks with containerized worlds, solvability review, and adversarial grader audits.
Tool use & interaction
Evaluations of multi-turn coordination, tool execution, state changes, and policy compliance.
- 01AgentBenchAgentBench · 2024Shows how performance varies across interactive environments, so one average can hide where an agent fails.
- 02GAIAMeta, Hugging Face & AutoGPT · 2023Introduces research-heavy, multimodal tool-use tasks and later illustrates the risks of leakage and saturation.
- 03τ-benchSierra · 2025Measures conversational tool use against hidden state and policy constraints with repeated trials and pass^k.
- 04τ²-benchSierra · 2025Models dual-control tasks in which an agent and a user both act on shared state to reach an outcome.
- 05ToolSandboxApple · 2025Accepts different valid action sequences while separately flagging prohibited actions.
- 06Berkeley Function Calling LeaderboardUC Berkeley · 2024Moves function-calling evaluation beyond syntax toward execution correctness, state, and multi-turn behavior.
Safety & operations
Evidence about prompt injection, harmful autonomy, long-horizon reliability, and operational risk.
- 01AgentDojoETH Zurich & Invariant Labs · 2024Evaluates indirect prompt injection in dynamic, stateful tool environments and measures the safety–utility tradeoff.
- 02InjecAgentUniversity of Illinois Urbana-Champaign · 2024Provides broad poisoned-tool-response coverage with stage-level analysis of extraction and exfiltration attacks.
- 03AgentHarmGray Swan AI & UK AI Security Institute · 2025Tests harmful multi-tool behavior, refusals, matched benign capability, and the effect of jailbreaks.
- 04Measuring AI Ability to Complete Long Software TasksMETR · 2025Uses human task time to build reliability curves and distinguishes median capability from high-confidence completion.
- 05RE-BenchMETR · 2024Compares agent and expert progress on machine-learning research work under equalized time and compute budgets.
- 06CyberSecEval 3Meta · 2024Separates cyber capability, uplift, autonomy, prompt injection, and guardrail tests into operationally meaningful phases.
THE BOTTOM LINE
Put the right system in the right world.
Measure what actually happened.
A useful eval does not promise certainty. Start with the decision; define the work, world, witnesses, and gate; repeat until the evidence is strong enough to act.