Evaluate an AI agent: traces, test cases and release decisions

Define the result before choosing a tool

Describe a task you can check: find the correct passage, draft an answer supported by it, or prepare an action for approval. Record the expected result, permitted data and tools, and conditions that require the agent to stop. A fluent answer is insufficient when the task includes an action.

Keep observability and evaluation separate. A trace helps explain the steps a run took. An evaluation checks those steps and the outcome against your criteria. Neither automatically enforces permissions or prevents a bad action.

• Include ordinary tasks, known failures, missing information, conflicting sources, unavailable tools and requests beyond the agent's permission.
• Keep a held-out set for the final comparison. Do not repeatedly tune on every case you will use to judge the change.
• For action-taking agents, inspect the final environment state and any unintended changes as well as the final message.

Capture enough evidence to explain a failure

Record a case identifier, application and model versions, allowed tools, relevant retrieved sources, tool results, elapsed time and the final output or state. Separate a wrong answer from a retrieval miss, tool failure, timeout or unauthorized action.

Before enabling tracing, inspect what leaves your system. Prompts, retrieved files and tool arguments can contain private data or credentials. Set redaction, retention, access and deletion rules; a self-hosted dashboard can still call remote models or evaluators.

• Start with deterministic checks for required fields, exact values and forbidden side effects where those checks fit the task.
• Use a written rubric for human review. If you add a model judge, compare its decisions with human labels and retain disagreements for review.
• Count failed, missing and timed-out runs explicitly. Record model and evaluator charges separately from hosting and maintenance costs.

Compare the same task before and after a change

Use the same case definitions, permissions, environment and scoring rubric for the baseline and candidate. Record any unavoidable difference in model availability or source data. Repeat variable tasks and keep the individual outcomes; an average can hide a serious failure.

Example arithmetic only: if 8 of 10 attempted cases pass, the pass rate is 8 / 10 × 100 = 80%. A timeout stays in the denominator. Report a forbidden action separately even if the overall pass rate rises. These figures are illustrative, not ASE or vendor measurements.

• Set acceptance criteria before examining the candidate's results, including any failure that blocks release regardless of the average score.
• Keep the smallest change that resolves the observed failure without unacceptable regressions, latency or cost.
• Retain the baseline and a recovery path. Turn confirmed production failures into new cases without exposing private customer material.

Know what a passing evaluation leaves unknown

A small test set can miss rare failures, new inputs, changed services and deliberate attacks. A model judge can share the same blind spots as the agent. An evaluation tool does not replace authorization checks, isolation, human approval or incident recovery.

Keep your current logging and a small local test harness if they already explain failures and let you compare changes. Add a platform when you need a specific capability, such as shared review, experiment tracking or trace search. More dashboards alone do not establish better results.

Choose a tool by the evidence you need

These are documented starting points, without a measured winner. Read the linked primary sources and test your own workflow.