How to Evaluate AI Agent Performance
Most teams deploying agents evaluate them the same way they evaluate chatbots, by looking at the final response and judging whether it is good. This misses most of what matters about agent performance. An agent that arrives at the correct answer by calling the wrong tools, making unnecessary API calls, or narrowly avoiding a dangerous action is not a well-performing agent, even though its output looks fine. Agent evaluation must be multi-dimensional, covering outcomes, trajectories, costs, and safety.
Step 1: Define Task Completion Metrics
The foundational metric for any agent is task completion rate: the percentage of tasks the agent completes successfully. This requires defining what "successful completion" means for each task type, which is less obvious than it sounds.
For a customer service agent, completion might mean: the customer's issue was resolved (verified by system state change), the resolution matched company policy, and no follow-up was needed. For a coding agent, completion might mean: the code passes all tests, introduces no new warnings, and the diff matches the scope of the requested change. For a research agent, completion might mean: the report covers all requested topics, cites verifiable sources, and contains no factual errors.
Beyond binary pass/fail, measure partial completion rate. An agent that resolves 70% of customer issues completely and makes meaningful progress on another 20% is substantially more useful than one that resolves 70% and fails completely on the other 30%. Partial completion metrics tell you whether the agent's failures are total (it could not do anything useful) or graceful (it did most of the work and identified what remains).
Measure failure categorization as well. An agent might fail because the task was genuinely impossible (the requested data does not exist), because it made an error in reasoning, because a tool failed, or because it hit a safety guardrail. Each failure category has a different remedy: impossible tasks need better routing, reasoning errors need prompt improvements, tool failures need infrastructure fixes, and guardrail triggers need policy review.
Step 2: Measure Trajectory Quality, Not Just Outcomes
Two agents can arrive at the same correct answer through very different paths. One might make 3 efficient tool calls. The other might make 15 calls, 10 of which were unnecessary, including calls to the wrong tools, redundant searches, and dead-end explorations. Both "completed the task," but the second agent wasted tokens, time, and potentially rate-limited API calls. Trajectory quality is what separates a reliable agent from a lucky one.
Trajectory metrics include: number of tool calls per task (lower is generally better for the same outcome), percentage of tool calls that contributed to the final answer (versus calls that were wasted), number of backtracking or retry events (which indicate the agent went down wrong paths), and whether the agent's reasoning at each step was logically connected to the previous step and the overall goal.
The most rigorous trajectory evaluation uses an LLM-as-judge approach. A separate model reviews the agent's full trace (every reasoning step, every tool call, every result) and evaluates whether each step was necessary, correct, and well-reasoned. This is expensive but catches quality issues that outcome-only metrics miss entirely. The LLM-as-judge evaluation guide covers how to set this up reliably.
A practical shortcut is to compare trajectories against expert traces. Have a human expert complete the same tasks, recording the steps they take, and then compare the agent's trajectory against the expert's. Where the agent diverges, investigate whether the divergence produced a worse outcome (a real problem) or just a different-but-equivalent path (not a problem). Over time, the expert traces become a benchmark that new agent versions are evaluated against.
Step 3: Track Cost and Efficiency Metrics
Agent costs are driven by the number of model calls, the size of the context window at each call, and the cost of external tool invocations. Tracking these metrics per task reveals optimization opportunities and catches regressions where a prompt change or model update causes the agent to use more iterations than before.
Key cost metrics to track for every task: total tokens consumed (prompt + completion), number of model calls (loop iterations), number of external tool calls (API requests, database queries), wall-clock time from task start to completion, and dollar cost (computed from token counts and model pricing). The cost analysis page provides formulas for computing per-task costs across different architectures.
Set cost budgets per task type. A customer lookup should not cost more than $0.05. A research report should not cost more than $2.00. If a task exceeds its budget, it either indicates an efficiency problem (the agent is doing unnecessary work) or a scope problem (the task is more complex than the budget assumes). Budget overruns are early warning signals that something in the agent's behavior has changed.
Compare cost-per-successful-task, not just cost-per-task. An agent that costs $0.50 per task but completes 95% of them successfully has an effective cost of $0.53 per successful completion. An agent that costs $0.30 per task but only completes 60% has an effective cost of $0.50. The cheaper agent is not actually cheaper when you account for the tasks it fails and the human labor needed to handle those failures.
Step 4: Evaluate Safety and Boundary Compliance
Safety evaluation tests whether the agent stays within its defined boundaries under both normal and adversarial conditions. This is not optional, it is the evaluation dimension that determines whether you can deploy the agent to real users without unacceptable risk.
Test boundary compliance with scenarios designed to push the agent outside its boundaries. Ask it to do things it should refuse (access data it is not authorized to see, take actions outside its scope, share confidential information). Ask it to handle requests that are edge cases (ambiguous permissions, borderline policy situations). Ask it to process adversarial inputs (prompt injection attempts, requests designed to confuse or manipulate it). Measure the rate at which it correctly refuses, correctly proceeds, or incorrectly crosses a boundary.
Tool misuse testing specifically evaluates whether the agent calls tools with parameters that violate constraints. If the agent has access to a database deletion tool with a scope limit of the current user's data, test whether it can be coerced into passing a parameter that deletes another user's data. If it has access to an email tool, test whether it can be convinced to send emails to addresses outside the approved list. The guardrails implementation guide covers how to enforce these constraints, and the agent security page covers the threat model in detail.
Safety metrics should have zero tolerance for critical violations and low tolerance for moderate violations. A critical violation, such as leaking customer data or executing an unauthorized financial transaction, should trigger an immediate deployment rollback. A moderate violation, such as generating a response that is off-topic or slightly outside brand guidelines, should trigger a prompt review. Track violations per 1000 tasks and trend them over time to ensure the rate is decreasing, not increasing.
Step 5: Build Automated Evaluation Pipelines
Manual evaluation does not scale. Once you have defined your metrics and created test scenarios, encode them into an automated evaluation pipeline that runs on every agent update, prompt change, or model version upgrade. This is the agent equivalent of a CI/CD test suite.
The evaluation pipeline should include three types of tests. Unit tests verify individual tool calls: given this input, does the agent call the right tool with the right parameters? These are fast, cheap, and catch regressions in basic functionality. Integration tests run full task scenarios end-to-end: given this customer question with this account state, does the agent reach the correct resolution? These are slower and more expensive but test the complete agent behavior. Adversarial tests attempt prompt injection, boundary violations, and edge cases: does the agent correctly refuse or escalate?
Use a golden test set of 50 to 200 tasks with known-good outcomes and trajectories. Run this test set on every change and compare results against the baseline. Any regression in task completion rate, any increase in average cost, or any new safety violation should block deployment until investigated. This is the same discipline as unit testing in software engineering, applied to agent behavior.
For evaluation tooling, the LLM evaluation pillar covers the major frameworks: DeepEval for Python-native evaluation with built-in metrics, Ragas for RAG-specific evaluation that applies to agentic RAG systems, LangSmith for LangChain/LangGraph agent tracing and evaluation, and Braintrust for comparative evaluation across model versions. Each has strengths for different parts of the agent evaluation pipeline.
Agent evaluation must be multi-dimensional: task completion rate tells you if the agent works, trajectory quality tells you if it works efficiently, cost metrics tell you if it is economically viable, and safety testing tells you if it is safe to deploy. Build an automated evaluation pipeline with golden test sets that runs on every change, and measure cost per successful task, not just cost per task.