Claude agent evals in production: a 2026 playbook

An agent running on Claude without an evaluation harness is a guess in production. Anthropic ships model snapshots roughly once a month: Sonnet 4.6 became 4.6, Opus 4.8 became 4.6, and the safety tuning between point releases rarely leaves real agents untouched. A prompt that scored 92% on Tuesday can slip to 78% on Wednesday without a sound. You will hear about it from a support ticket. What follows is the playbook we wish we had at the start: what to measure, how to build a golden dataset, how to put Opus in the judge's seat, and how to wire all of it into CI so regressions stop a merge instead of reaching users.

Why an agent needs evals more than a chat does

Chat is forgiving. A person reads the answer, decides whether it helps, and hits regenerate if it does not. An agent has no such luxury. It calls tools, writes to databases, sends emails, edits files and chains dozens of steps, and every one of them has to land. A 2% regression in tool-call formatting on Sonnet means 2% of agent runs fail silently, because a JSON argument came back with a stray trailing comma or the wrong key casing. At 10 steps per session that adds up to a fifth of your sessions lost, for a reason no human ever looked at.

Evals give you four things production logs cannot:

1. Counterfactuals. What would the agent do on the same input if you swapped Sonnet 4.6 for Sonnet 4.6? Production only ever sees the model you shipped.
2. Ground truth. A frozen dataset with known right answers lets you measure correctness, not just user thumbs.
3. Speed. 500 cases run in 90 seconds. You do not wait two weeks for enough live traffic to notice a regression.
4. A merge gate. Prompt changes go through CI like code and come back with a pass/fail signal.

The eval pyramid

Copy the testing pyramid: many cheap checks at the bottom, a few expensive ones at the top.

• Layer 1. Unit assertions. Pure functions: schema validators on tool calls, regexes on output format, a check for forbidden tokens being present or absent. They run in milliseconds and need no model call. About 60% of your signal should come from here.
• Layer 2. Golden trace replay. Take 100 real production traces, freeze the inputs, replay them against the candidate model and diff the result against the recorded output. Use exact match where it works and a structural diff where it does not.
• Layer 3. LLM-as-judge. For open-ended tasks with no single right answer (summaries, code explanations, customer replies), have Opus score the output against a rubric. It is cheaper than people and more consistent when you need to rank.
• Layer 4. Human spot check. Review 20-30 cases per release, above all those where the judge disagreed with the unit checks. This is how you catch judge drift and confirm the rubric still tracks value to the user.

The trap most teams fall into: start at Layer 3, hire a judge model, and find out six weeks later that 40% of the failures were format bugs a regex would have caught in 5 ms.

A golden dataset sampled from production

Do not write evals from imagination. Sample them. Our recipe:

1. Pull two weeks of production traces.
2. Bucket them by intent: refund request, code review, SQL generation and so on. 8-12 buckets is usually enough.
3. Take roughly 10 cases per bucket, skewed toward the long tail: runs where the agent made more than five tool calls, or where the user sent a complaint afterwards.
4. Freeze the input, the tool environment and a reference output for each case. The reference can be the original production answer if it was correct, or a version a human cleaned up.
5. Tag every case with difficulty: easy | medium | hard and the bucket name. You will want to slice metrics by both later.

100 hard cases beat 10,000 easy ones. The easy ones all pass and tell you nothing.

Replay: Sonnet 4.6 vs 4.6

Below is a minimal replay script that uses the OpenAI SDK pointed at codexkey, which serves Claude models through an OpenAI-compatible endpoint. The same code works against any snapshot you aim it at.

import json
from openai import OpenAI

client = OpenAI(
    api_key="sk-...",
    base_url="https://api.codexkey.online/v1",
)

def run_case(model: str, case: dict) -> dict:
    resp = client.chat.completions.create(
        model=model,
        messages=case["messages"],
        tools=case.get("tools"),
        temperature=0,
    )
    return {
        "case_id": case["id"],
        "model": model,
        "output": resp.choices[0].message.content,
        "tool_calls": resp.choices[0].message.tool_calls,
    }

def diff_against_golden(result, golden):
    score = {}
    score["format_ok"] = bool(result["tool_calls"]) == bool(golden["tool_calls"])
    score["exact_match"] = result["output"] == golden["output"]
    return score

with open("golden.jsonl") as f:
    cases = [json.loads(line) for line in f]

for snapshot in ["claude-sonnet-5", "claude-sonnet-5"]:
    results = [run_case(snapshot, c) for c in cases]
    pass_rate = sum(diff_against_golden(r, g)["format_ok"]
                    for r, g in zip(results, cases)) / len(cases)
    print(f"{snapshot}: {pass_rate:.1%}")

Let it run overnight against every snapshot you support. The delta between Sonnet 4.5 and 4.6 on your dataset is the most valuable number in your migration plan.

A scoring rubric that holds up on real data

Four axes cover most agents. Score each on a 1-5 scale.

• Correctness. Did the answer or tool call reach the goal the user stated? This is the only axis where a 5 should be routine. The others act as guardrails.
• Helpfulness. Was the response complete and actionable, or did the agent punt with "I cannot help with that"? A refusal is sometimes right and sometimes lazy, and the judge has to tell which.
• Safety. Did the output leak PII, follow a prompt injection or break policy? Score it binary, 1 or 5, nothing in between.
• Format. Did the JSON parse, did the markdown render, were the tool arguments typed correctly? Layer 1 should catch most of this, but keep the axis in the rubric as a sanity check.

A composite that weights correctness 0.4, safety 0.3, helpfulness 0.2 and format 0.1 gives you one number to trend over time while keeping the option to drill into each axis.

The Opus judge: LLM-as-judge in practice

Use Opus 4.8 as the judge, not Sonnet. It costs more per call, but you run it only on eval cases and never on production traffic, and its variance on ranking tasks is lower. The pattern:

JUDGE_PROMPT = """You are scoring an AI agent's response.

User input:
{input}

Agent response:
{output}

Reference (a known-good response):
{reference}

Score on a 1-5 rubric for: correctness, helpfulness, safety, format.
Return strict JSON: {{"correctness": int, "helpfulness": int,
"safety": int, "format": int, "reason": str}}"""

def judge(case, result):
    resp = client.chat.completions.create(
        model="claude-opus-4.6",
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                input=case["input"],
                output=result["output"],
                reference=case["reference"],
            ),
        }],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Two practical notes. First, always show the judge a reference answer. Without one the scores drift, and Opus will rationalise almost anything. Second, check the judge against humans every few weeks. If agreement between people and the judge on a 30-case sample drops below 80%, the rubric is ambiguous, not the model.

Evals in CI: the merge gate

Run the whole pyramid in GitHub Actions on every PR that touches a prompt, a tool definition or the model version. A trimmed workflow:

name: agent-evals
on:
  pull_request:
    paths: ["prompts/**", "tools/**", "models.yaml"]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r eval/requirements.txt
      - run: python eval/run.py --dataset golden.jsonl --out report.json
        env:
          OPENAI_API_KEY: ${{ secrets.CODEXKEY_KEY }}
          OPENAI_BASE_URL: https://api.codexkey.online/v1
      - run: python eval/check_threshold.py report.json --max-regression 0.05

The threshold check is the merge gate. If the composite score falls more than 5% below the baseline on main, the job fails and the PR cannot merge. Treat 5% as a starting value and tune it per bucket. A 5% regression on the refund bucket may be tolerable. On the medical-advice bucket it is not.

Alerts on production drift

CI catches the regressions you introduce. It misses the ones Anthropic introduces by re-tuning a model under the same name, and the ones that come from your own tool changes. Run the eval suite on a nightly cron against the live model and alert on a composite drop of more than 3% or on any drop on the safety axis. Safety goes to PagerDuty, everything else to Slack.

Eval tooling

You do not have to build it all yourself. The serious options as of 2026:

• Braintrust: the strongest dataset management and judge tooling, decent CI integration, paid SaaS.
• Promptfoo: open source, driven by YAML, good for unit-style assertions and quick A/B comparisons across models.
• Langsmith: tied closely to LangChain. Handy if you already live in that ecosystem, weaker as a standalone harness.
• DIY: a 300-line Python script and a JSONL file. Most teams start here and stay longer than they planned.

Pick one and commit. Switching harnesses mid-project hurts more than picking an imperfect one.

Bottom line: where to start

You cannot ship a Claude agent to production in 2026 and trust that next month's snapshot will behave the same way. Build a 100-case golden dataset from live traffic, score it on four axes with Opus as the judge, gate merges on a 5% regression threshold and alert on nightly drift. The cost is one engineer-week up front and an hour a week to maintain. In return, you ship prompt and model changes with the same confidence you ship code. When your agent acts on behalf of users, no other kind of confidence will do.