How to Evaluate an AI Agent Without Fine-Tuning Anything

I read ten of an agent's responses. They look fine. I ship. That's the whole test most of us run, and ten examples say nothing about the other several thousand a real agent handles in a week.

Grade it with a second model call. The core of it is three steps the checklist below walks through: write a rubric with named criteria, hand-grade a batch of responses yourself, then run the judge on that same batch and measure agreement with Cohen's kappa. Run your hardest task five to eight times before you call the agent reliable.

This guide starts once you have an agent worth grading: one that retrieves from your own documents with a similarity floor that decides when to refuse in place. If the job didn't need an agent at all, that test comes first.

What is an LLM judge?

An LLM-as-judge is a second model call. You pass it the first model's response, a rubric, and sometimes the source material the response should be drawn from. It hands back a grade.

A bare "is this good?" prompt won't return the same verdict twice: two runs on the same response can land on different answers for different reasons.

Name three to five criteria in your rubric, the way you'd brief a new hire:

• Correctness. Does the answer match the source document, not just sound plausible?
• Completeness. Does it cover every part of the question, not only the easiest part?
• Tone. Does it match the policy or script the business already uses, word for word where that matters?

Airbnb built exactly this for a faithfulness evaluator, an LLM judge checking whether an assistant's answer followed from the source it cited. Their first version agreed with a human product manager 78% of the time, called "not good enough" internally.

After they rewrote the rubric and added few-shot examples of what a pass and a fail look like, agreement reached 88%. They measure it with Cohen's kappa or Krippendorff's alpha, not raw percent match, because either one corrects for the agreement two graders would get by flipping a coin.

Their own process: build a golden set of 50 to 100 examples, and make sure it includes bad ones on purpose. Run the judge against that set, measure agreement, and target the high 80s to 90s. If agreement is low, fix the rubric and the few-shot examples, not the underlying model.

Why grade a sample of traffic instead of everything?

Airbnb doesn't run its judge on every response. It samples 5% of live, de-identified traffic daily.

Airbnb's 5% slice stays cheap enough to run every day, indefinitely. Grading every response doesn't: a judge you stopped running two months ago tells you nothing about what shipped last Tuesday.

How many judges does it take?

Airbnb's own line: "3–5 well-calibrated LLM-as-judge evaluators beat 20–30 noisy ones."

A calibrated judge is one whose agreement with a human has been measured against a golden set, the way Airbnb measured theirs against the PM's own grades. If you never check a judge against a human, you don't know what the grade is worth.

Adding more automated graders to a system that has never measured whether even one of them agrees with a human doesn't buy reliability. Twenty unchecked judges all agreeing with each other still disagree with your own hand-grades.

What biases does a judge carry?

The paper behind the "LLM judge" idea, MT-Bench, measured the judges as hard as it measured the models being judged. GPT-4 as judge agreed with human graders 85% of the time, higher than the 81% agreement two human graders had with each other.

The same paper then measured where a judge goes wrong.

What MT-Bench measured	The number	The fix
Position bias	Swap the order of the same two answers, and GPT-4's own verdict held only 65% of the time, with a 30% pull toward whichever answer it read first	Grade every pair both ways and flag a verdict that flips when the order does
Verbosity bias	A wordier, wrong answer beat a concise, correct one 91.3% of the time when the judge was a weaker model (Claude-v1, GPT-3.5), against only 8.7% of the time for GPT-4	Choose a strong judge model, and don't assume a rubric that behaves well on one model behaves the same on a cheaper one
No reference answer	Grading math questions against no reference answer, the default prompt failed 70% of the time	Give the judge the correct answer to grade against; failures dropped to 15%

The 85% agreement figure is GPT-4's. A weaker judge model failed the verbosity test on more than 9 out of 10 tries, which means a rubric calibrated on GPT-4's judgment doesn't transfer to a cheaper one without re-checking.

Why isn't one passing run enough?

Sierra's τ-bench tested airline and retail support agents built on GPT-4o-class models against realistic multi-step tasks with real policy constraints. The agents succeeded on under 50% of tasks run once. That single-run number is pass@1, and it's what most of us report by accident, because we only ran the task once.

pass@k runs the same task k times and counts how many of those runs succeeded. Sierra ran the same retail task eight times in a row, pass@8, and the agent passed fewer than 25% of the time.

τ-bench's own agent looked fine on any individual attempt and still failed most of the time it was asked to do the identical thing again.

Run this checklist this week

1. Write a rubric with 3 to 5 named criteria, not a single yes-or-no question.
2. Hand-grade 20 to 30 real responses yourself. This is the answer key your judge gets measured against.
3. Run your judge on that same set of 20 to 30, and check Cohen's kappa against your own grades.
4. If kappa comes back below the high 80s, go criterion by criterion: is there a written example of a passing response and a failing one? That's usually the gap. Add one of each and re-run. Airbnb's first version sat at 78%; a rewritten rubric reached 88%.
5. Run your hardest task 5 to 8 times and report pass@k, not the single run you happened to check.

Quick recap

• An LLM judge is a second model call. It reads the first model's response against a rubric with named criteria and returns a grade.
• Measure the judge against your own hand-graded examples with Cohen's kappa or Krippendorff's alpha. Raw percent agreement overcounts what would happen by chance.
• Sample a fixed slice of traffic daily rather than grading everything; Airbnb runs its judge on 5%.
• A few well-calibrated judges beat a pile of judges nobody checked.
• A judge has known failure modes: position bias, verbosity bias, and math failures with no reference answer to grade against.
• One successful run is pass@1. Run your hardest task 5 to 8 times and report pass@k instead.

Start Here

The intake at daisyguti.ai/work-with-me is about nine questions and takes a few minutes, held as a live conversation with an AI. It turns your answers into a brief with the specifics that matter: what you're trying to solve, what you've already tried, what a working version looks like to you.

I read that and reply with whether to start on the agent itself or on the evaluation layer you'd eventually put on top of it.

Sources

1. Airbnb Tech Blog, "Eval-driven development: Lessons from evaluating GenAI at scale" - https://airbnb.tech/ai-ml/eval-driven-development-lessons-from-evaluating-genai-at-scale/ - the faithfulness evaluator's 78% to 88% agreement, the golden-dataset process, the 5% daily sampling, and "3-5 well-calibrated LLM-as-judge evaluators beat 20-30 noisy ones."
2. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" - https://arxiv.org/html/2306.05685 - the 85%/81% agreement figures, the position-bias and verbosity-bias measurements, and the reference-guided grading result (70% to 15%).
3. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" - https://arxiv.org/abs/2406.12045 - the under-50% pass@1 and under-25% pass@8 (retail) figures for GPT-4o-class agents.