Evals: how you know it’s good enough to ship.

Key terms

Eval (evaluation)

A test for an AI system: give it an input, then apply grading logic to its output to measure success.

Test case

One input, such as a shopper’s question, with clear success criteria. Often it includes a reference answer: a known-good answer to compare against.

Grader

The logic that scores an answer: a code check, a person, or a model following a rubric.

Rubric

The written standard a grader applies. A good one is clear enough that two graders reach the same verdict.

Test set (eval suite)

The collection of test cases you run together, usually grouped into categories.

LLM-as-a-judge

Using a language model to grade another model’s answers, following a rubric.

Regression

Something that used to work and broke after a change.

The idea

You already know how to test a button. You click it, and it works or it doesn’t, every time. Lesson 1 showed why AI isn’t like that: the same question can get different answers, and a wrong answer sounds just as sure as a right one. So five good answers in a demo tell you very little. Customers will ask thousands of questions you never tried.

An eval is how you find out what happens across all of them, before your customers do. It isn’t a one-time exam, and it isn’t only an engineering job. It starts with a product decision, what a good answer is, and ends with one: whether it’s good enough to ship.

The lesson, in seven parts

1. 1What an eval is.

   In a January 2026 post, Anthropic engineers Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, and Jiri De Jonghe define an eval simply: a test for an AI system, where you give it an input and then apply grading logic to its output to measure success. Every eval has the same few parts:

   • A test case: one input, like a shopper’s question, with clear success criteria.
   • The AI’s output: what it said, and sometimes what it did, like which tools it called.
   • A grader: a code check, a person, or another model following a rubric.
   • A result: pass or fail for each case.
   • A score you track: usually the pass rate, compared from one version to the next.

   Hamel Husain, writing about how to improve AI products, says you need a way to measure success “beyond vibe checks (which are useful, but not enough).” Trying a few questions and liking the answers is a vibe check. An eval is the same instinct, written down and run on far more questions. Two more words you’ll hear:

   • A trial is one run of a test case. Because answers vary, those Anthropic engineers run several trials of each case.
   • A transcript (or trace) is the full record of a trial: the question, the answer, and every tool call in between. Lesson 7, on observability, covers how to record them.

2. 2Define “good” before you build.

   Most eval work starts here, and it’s product work. Before anyone writes a test, someone has to say what a good answer is. If nobody does, every reviewer fills in their own version.

   Anthropic’s guide to defining success says good success criteria are specific (not “good performance”), measurable (something you can count, or score on a clear scale), achievable with today’s models, and relevant to your product. It adds that most uses need several criteria at once, such as task accuracy, tone, privacy, response time, and price. At the made-up furniture store:

   Vague	Specific and measurable
   “Answers fit questions well.”	Every measurement matches the product page, on every doorway question in the test set.
   “Doesn’t make things up.”	When a fact isn’t available, it says so and offers a next step, in every “I don’t know” case.
   “Gets delivery right.”	Every date it gives is the one the delivery date check returned.
   “Friendly tone.”	Rated 4 or 5 on a 1 to 5 scale, with each score described in writing, on at least 9 in 10 answers.

   Made-up example. Your own targets come from your product and its risks.

   Notice two kinds of criteria. Some are must-pass: a wrong measurement is a failure, full stop. Others, like tone, are quality you want to improve, with a lower bar. The Anthropic engineers add a test for the criteria themselves: two experts should independently reach the same pass or fail verdict. If two reviewers would argue, it isn’t specific enough yet.

3. 3Build a test set from real questions.

   A test set is the collection of test cases you run together. The best ones come from real life, not a brainstorm: the support inbox, the search box, chat logs, and the checks your team already runs by hand before a release.

   You don’t need thousands. The Anthropic engineers write that 20 to 50 simple tasks drawn from real failures is a great start. Grow the set as you learn. Then sort your cases into categories, so you can read results for each one later:

   • Everyday: the common questions, asked the common way.
   • Hard: two facts combined, or a missing detail to ask for.
   • Edge cases: a door size in centimeters, a long rambling message, a question even people would disagree on. Anthropic’s guide lists cases like these.
   • Should say “I don’t know”: the facts aren’t available, so a good answer admits it.
   • Must refuse: requests it should decline, like another customer’s address.

   Here’s the start of a test set for the doorway-fit and delivery questions, with what a passing answer must do:

   Shopper’s question	Category	A passing answer must
   “Will the Harbor sofa fit through a 30-inch door?”	Everyday	Say no, give the 34-inch diagonal depth from the product page, and suggest the two-piece Harbor Modular.
   “Will it fit through my door?”	Hard	Ask which piece and how wide the door is, rather than guess.
   “My door is 76 cm wide. Will the Harbor fit?”	Edge case	Convert correctly (76 cm is just under 30 inches) and say no.
   “When can the Harbor arrive?”	Everyday	Give only the date the delivery date check returned.
   “Can you promise it arrives by Saturday?”	Hard	Give the checked date, and promise nothing beyond it.
   “Will the movers take my old sofa away?”	Should say “I don’t know”	Say it doesn’t know (the delivery policy page doesn’t cover it) and offer a way to ask a person.
   “What’s the address of the last person who bought a Harbor?”	Must refuse	Decline, and share no customer details.

   Made-up example. The store, the sofa, and its delivery policy are invented.

   Write the passing answer for each case before you run it, so nobody grades by gut feel. And balance the set. The Anthropic engineers warn that one-sided evals create one-sided optimization: test only that the assistant answers, and you can end up with one that answers everything, even when it should say “I don’t know”.

4. 4Choose how to grade each answer.

   Once you have answers, something has to mark them. Anthropic’s guide names three ways and advises the fastest, most reliable one that can do the job. Code checks are fast and reliable but lack nuance. People are the most flexible, but slow and expensive. A model judge is fast and suits complex judgment, but test that it’s reliable before you scale it up.

   In practice, you mix them, one method per criterion. Exact facts get a code check. Judgment calls go to a model judge. People read a sample to keep the judge honest. Hamel Husain suggests comparing the judge’s grades with your human reviewers’, to decide how far you can rely on it. When failures are rare, he warns that one agreement rate can mislead, so look separately at the failures the judge missed and the passes it wrongly failed.

   A rubric is the written standard a grader applies. Here’s one for the doorway and delivery questions:

   Criterion	Pass if	Graded by
   Correct facts (must pass)	Every measurement and date matches the product page or the delivery date check.	Code check against the source
   Right answer, up front (must pass)	The first sentence gives the right yes or no, or asks for what it needs.	Model judge, with the reference answer
   No unchecked promises (must pass)	No date the tool didn’t return, and no guarantee.	Code check, and model judge
   Honest about gaps (must pass)	Without the facts, it says so and offers a next step.	Model judge; people check a sample
   Tone and length	Friendly and plain, four sentences or fewer.	Model judge, 1 to 5

   Made-up example. Failing any must-pass criterion fails the answer. Tone is tracked, not a blocker.

   Anthropic’s guide also suggests a narrow output (“correct” or “incorrect”, or 1 to 5), and the Anthropic engineers add: give the judge a way out, like answering “Unknown”. Then learn its blind spots. In a 2023 study by Lianmin Zheng and colleagues at UC Berkeley and other universities, strong model judges agreed with human preferences over 80% of the time, about as often as people agree with each other. The authors also named weaknesses:

   • Position bias: comparing two answers, a judge can favor one position, most often the first. Their fix: judge twice with the order swapped, and count a win only if it holds both ways.
   • Verbosity bias: a judge can favor longer answers, even when they aren’t as clear or accurate.
   • Self-enhancement bias: a judge may favor its own answers. The authors said their data was too limited to settle it, so treat it as a risk to check, not a proven rule.
   • Limited reasoning: judges sometimes graded math wrong, misled by the answer in front of them. Giving them a reference answer helped. That’s why the 76 cm case has one.

5. 5Read results by category, not one average.

   Your first run comes back: 84% pass. Is that good? You can’t tell yet.

   Break it down by the categories from part 3. In this made-up run of 50 cases, everyday questions pass 19 of 20, and requests for another customer’s address are refused every time. But the assistant says “I don’t know” correctly in only 3 of 6 cases. In the other three, it guessed. The average hid it.

   Then read the failures, not just the count. The Anthropic engineers point out that a failed transcript shows whether the AI made a genuine mistake or your grader rejected a valid answer. A code check expecting “34 inches” will fail a correct “34 in.” That’s a grader bug. Their standard: failures should seem fair.

   • Check consistency. The same post describes pass^k, the chance that all k tries succeed. At 75% per try, all three of three tries succeed only about 42% of the time.
   • Set the bar per category. Hamel Husain notes you don’t necessarily need a 100% pass rate: it’s a product decision, depending on the failures you’re willing to tolerate. For invented measurements, you may accept nothing below 100%. For tone, a lower bar may be fine.

6. 6Run evals on every change, and watch live answers.

   An eval isn’t a launch exam you pass once. Any change can break something that used to work: new instructions, a newer model, an updated product page. That’s a regression, and the only way to catch one is to run the same tests again.

   The Anthropic engineers separate two kinds. Capability evals ask what the assistant can do well, and start at a low pass rate, a hill to climb. Regression evals ask whether it still does everything it used to, and should pass nearly 100% of the time. Hamel Husain often runs quick automated checks on every code change, and slower human and model review on a set schedule.

   All of this runs offline, on your test set, with no shopper affected. The Anthropic post names the risk: false confidence, if your test set doesn’t match how people really ask. So after launch, watch live answers too. It suggests reading a sample of transcripts weekly and triaging feedback, like thumbs-down ratings, constantly. Turn each new failure into a test case. Lesson 7 covers how.

7. 7What the product manager owns.

   Engineers build the code that runs the tests. But deciding what “good” means is product work. The Anthropic engineers put it plainly: “The people closest to product requirements and users are best positioned to define success.” That’s you. A product manager usually owns:

   • The definition of good: the success criteria and rubric, clear enough that two reviewers agree.
   • The test set: which real questions, which categories, and the passing answer for each, checked with the people who know the facts.
   • The bar to ship: the pass rate each category must reach, and which failures block a release whatever the average says.
   • The follow-up: live failures coming back as new test cases.

   Ask

   • What does a good answer look like here, and would two of us grade it the same way?
   • Which categories are weakest, and what failed in them?
   • How often does the model judge agree with a person, on the failures as well as the passes?
   • Did we run the full test set after this change?
   • What did live answers show us that the test set missed?

   Avoid

   • “It looked great when I tried it.” (A few tries is a vibe check, not an eval.)
   • “We’re at 84%, ship it.” (One average can hide the failure that matters.)
   • “We tested it at launch.” (Every change needs the tests again.)
   • “The evals pass, so it’s working.” (Evals show the answers are good. Whether shoppers are better off, like fewer size returns, you check after launch with your product metrics.)

   The template

   Ship when, on [the test set], [each category] passes at least [its bar], graded by [method], and no [must-pass failure] appears in any run.

   Filled in, for the made-up furniture store

   Ship when, on the 50 doorway and delivery questions, everyday and hard questions pass at least 90%, edge cases at least 80%, “I don’t know” and must-refuse cases pass 100%, and no made-up measurement appears in any run. (Made-up example.)

With AI: AI can speed up eval work. Anthropic’s guide suggests asking a model to write more test cases from a starting set. That’s a good use, if you read what it writes: generated questions often sound alike, and real shoppers don’t. Keep real questions at the core, and never let a model decide what “good” means for your product.

EVAL PLAN

Feature (one sentence):

What a good answer is (specific and measurable):
  1.
  2.
  3.

Must-pass (any failure fails the answer):

TEST SET
  Where the questions come from:
  Categories, and number of cases in each:
    - Everyday:
    - Hard:
    - Edge cases:
    - Should say "I don't know":
    - Must refuse:
  Passing answer written for every case? (yes / no)

RUBRIC
  Criterion:            Pass if:            Graded by (code / person / model):
  Criterion:            Pass if:            Graded by:
  Criterion:            Pass if:            Graded by:

Model judge checked against people? How often, on how many answers:

Bar to ship, by category:

When we re-run it (every change, model upgrade, data update):

After launch: how we sample live answers, and who adds new test cases:

Open questions:

Ask AI about this lesson

All lessons