Freeze the question before comparing answers

An agent can produce an impressive trace while failing the task. Start with a strong direct baseline, a versioned set of representative tasks, fixed permissions and an explicit success rule. Include held-out cases so repeated prompt or harness changes do not overfit the examples used during development. Anthropic's agent-evaluation guide emphasizes task, environment and grader design.

Record the outcome alongside the trace: files changed, tests run, tool receipts, latency, model calls, human interventions and recoveries. A final answer is one observation, not the whole result. The useful cost metric is total operating cost divided by independently validated successes; report human review time separately rather than hiding it inside a score.

Run one comparison that can change a decision

Worked example

For a hypothetical repository assistant, freeze 20 issue-to-patch tasks with acceptance tests and permission limits. Run a direct prompting baseline and the proposed agent on the same tasks. For each attempt, store the patch, test log, external actions, elapsed time and reviewer minutes. Independently grade whether the patch solves the issue without unauthorized effects. A fluent explanation alone does not pass.

05What a comparison must show
Task outcomeAccepted artifact and tests
Operating costCalls, latency and retries
Human effortReview and repair minutes
Safety effectsDenied actions and receipts

Compare the baseline and candidate on the same frozen tasks before a release decision.

Original evaluation scorecard informed by agent evaluation guidance. It names evidence to collect; it does not show measured results.

Separate local smoke tests from release evidence. A change can pass unit tests and still fail because the browser state is wrong, an external API returned an uncertain receipt, or a human had to repair every result. Add adversarial inputs, timeouts, stale data and revoked permissions to the suite. Use fault injection where a production effect depends on recovery behavior.

Use three kinds of test evidence

First, deterministic tests check rules, permissions and expected software behavior. Second, agent-path evaluations inspect tool choice, arguments, plan adherence and wasted steps; DeepEval's agent metrics name several of these dimensions. Third, an independent acceptance check asks whether the task was actually completed without an unauthorized effect. A model judge can help triage traces, but it cannot certify an unobserved outcome.

Practical setup

Keep each test case's input, environment snapshot, expected artifact and grader version together. Use a test-management system such as QMetry to track cases and runs if useful; it is infrastructure for organizing evidence, not an agent benchmark. EPAM's testing-pyramid discussion argues that real model and tool integration tests deserve attention alongside unit tests. Measure that tradeoff in your own environment.

When a case fails, a distributed trace can reveal a slow or erroneous tool call, but the trace is diagnostic evidence rather than a pass condition. See the OpenTelemetry waterfall example and the specification-to-release workflow.

Keep the improvement loop inspectable

SIFT uses pairwise LLM judgments and ranking to prioritize revised agent harnesses. Those judgments are a search aid. Executed benchmark tasks still determine whether a candidate improved. This distinction generalizes: a judge, reward model or search score may guide development, but final acceptance needs independent task evidence.

In operation, retain enough run state to explain what happened and recover safely: versioned prompts and tools, task IDs, checkpoints, pending approvals, receipts and an audit trail. Keep a separate lifecycle for improving the agent itself—plan, build, evaluate, deploy—from the lifecycle of executing one user's task.

Read the original work

These sources support the specific concepts above. Their task settings and publication status matter when applying a result.

An agent under test is compared with a golden dataset using an evaluation engine for trajectory and response metrics.
Figure. Google ADK’s example compares agent behavior with expected tool use and answers. It illustrates one evaluation implementation. Source: Google Codelabs · ADK evaluation · CC BY 4.0.
Next: What language models learn