Lab / Explainer / 4 steps

No. 10Explainer7 min

How do you know it got better?

Grade AI answers yourself, write the rules down, compare two versions and read what got worse.

Short version

Keep a small fixed set of real questions, write down what a good answer must do, rerun it on every change, and read the cases that got worse before you look at the average.

The most common way to test an AI feature is trying five questions and going with a feeling. It works until the next change quietly breaks something. An eval is the product manager's version of a test: a fixed set of questions, written rules, and a comparison. Everything here is invented.

01 · Your gut

Grade six answers yourself.

A support assistant for a plant webshop gave these answers. Mark each as pass or fail. My grade turns over as you go, and the tally circles every answer we grade differently.

  1. 1Customer"My order has not arrived, it has been 5 days."

    AssistantSorry about the wait. Your order shipped on Tuesday and tracking shows delivery tomorrow before 18:00; here is the tracking link.

    Your grade for answer 1

    My grade: pass. Uses the order data and gives a next step.

  2. 2Customer"Can I return a plant that died?"

    AssistantPlants are living products, so unfortunately we cannot accept returns.

    Your grade for answer 2

    My grade: fail. Wrong: the shop has a 30-day plant guarantee, so the customer loses money they are owed. And it leaves them with no next step.

  3. 3Customer"Do you ship to Belgium?"

    AssistantGreat question! We love our Belgian customers and we are always happy to help. Shipping to Belgium is possible. Many customers there order every month. Let us know if you have other questions!

    Your grade for answer 3

    My grade: fail. Friendly, but five sentences and no price, delivery time or next step. Fails the length rule.

  4. 4Customer"Is the monstera safe for cats?"

    AssistantNo, monstera is toxic to cats. Put it out of reach, or choose a calathea, which is safe for pets.

    Your grade for answer 4

    My grade: pass. Correct, short, and offers an alternative.

  5. 5Customer"Cancel my order 4411."

    AssistantDone, order 4411 is cancelled.

    Your grade for answer 5

    My grade: fail. The assistant cannot cancel orders, so the order is still open; it should hand over to the team. A broken promise, and no word on the refund.

  6. 6Customer"Your app is terrible."

    AssistantSorry to hear that. What went wrong? I will pass it on to the app team today.

    Your grade for answer 6

    My grade: pass. Short, kind, asks for details, realistic promise.

Pass or fail, per answer

123456 You Me

You: 0 passMe: 3 of 6 pass

  • pass
  • fail
  • mine, until you grade it

I pass 3 of 6 (50%). With JavaScript on, you grade them too.

Show as text
#Customer asksAssistant answersMy gradeWhy
1My order has not arrived, it has been 5 days.Sorry about the wait. Your order shipped on Tuesday and tracking shows delivery tomorrow before 18:00; here is the tracking link.PassUses the order data and gives a next step.
2Can I return a plant that died?Plants are living products, so unfortunately we cannot accept returns.FailWrong: the shop has a 30-day plant guarantee, so the customer loses money they are owed. And it leaves them with no next step.
3Do you ship to Belgium?Great question! We love our Belgian customers and we are always happy to help. Shipping to Belgium is possible. Many customers there order every month. Let us know if you have other questions!FailFriendly, but five sentences and no price, delivery time or next step. Fails the length rule.
4Is the monstera safe for cats?No, monstera is toxic to cats. Put it out of reach, or choose a calathea, which is safe for pets.PassCorrect, short, and offers an alternative.
5Cancel my order 4411.Done, order 4411 is cancelled.FailThe assistant cannot cancel orders, so the order is still open; it should hand over to the team. A broken promise, and no word on the refund.
6Your app is terrible.Sorry to hear that. What went wrong? I will pass it on to the app team today.PassShort, kind, asks for details, realistic promise.

I pass 3 of 6.

fig. 1 Grade by feel, then compare. Every answer we grade differently is circled: a gut feeling is a test set of one person on one day.

Your gut is a test set of one person on one day. Useful to start, not enough to decide.

02 · The rubric

Write down what a good answer must do.

Most disagreements come from rules nobody wrote down. Is a friendly but long answer a pass? Here are three rules. Pick an answer and watch it go past them: one no is enough to fail.

The answer

3Customer"Do you ship to Belgium?"

AssistantGreat question! We love our Belgian customers and we are always happy to help. Shipping to Belgium is possible. Many customers there order every month. Let us know if you have other questions!

Three written rules

  1. R1 Correct according to our policy and the order data. yes
  2. R2 Promises nothing the assistant cannot do. yes
  3. R3 At most three sentences, and a clear next step. no: five sentences, and no next step

The result

FailOne no is enough.

All six against the rules

AnswerR1R2R3RulesYou
1 Late order yesyesyes pass ?
2 Dead plant noyesno fail ?
3 Belgium yesyesno fail ?
4 Monstera yesyesyes pass ?
5 Cancel order nonono fail ?
6 Bad app yesyesyes pass ?
Yes, of 64533 pass

You: your grades from fig. 1. Grade them there to compare.

Show the rules and all six results as text
  1. R1: Correct according to our policy and the order data.
  2. R2: Promises nothing the assistant cannot do.
  3. R3: At most three sentences, and a clear next step.

A next step tells the customer what happens next or what they can do now.

AnswerR1R2R3Result
1. My order has not arrived, it has been 5 days.yesyesyesPass
2. Can I return a plant that died?no: the shop has a 30-day plant guaranteeyesno: no next stepFail
3. Do you ship to Belgium?yesyesno: five sentences, and no next stepFail
4. Is the monstera safe for cats?yesyesyesPass
5. Cancel my order 4411.no: the order data says it is still openno: it cannot cancel orders: the team does thatno: no next stepFail
6. Your app is terrible.yesyesyesPass
fig. 2 Write the rules down. One answer goes past three rules and a single no fails it, so anyone who applies them gets the same 3 of 6. Where your grade from fig. 1 differs, it is underlined.

Three written rules turn a feeling into something two people, or a machine, can apply the same way.

03 · Versions

Compare versions, then read what got worse.

You change the prompt. On the 50-case set, version B passes more cases than version A. Ship it? Look at the cases that went from pass to fail first: the same case sits in the same spot in both grids.

Version A

41 of 50 pass (82%)

Version B

44 of 50 pass (88%)

The averages

A

41

B

44

B looks better: 3 more cases pass, 6 points up.

Case by case

6 fixed

3 broke

41 + 6 fixed − 3 broke = 44. Also 38 stayed a pass and 3 stayed a fail.

  • pass
  • fail
  • fixed in B
  • broke in B

The 3 that got worse, to read before you ship

  1. 1"Can I pay with a gift card and a bank card together?"

    A, passYes: at checkout, choose split payment.

    B, fails R1: split payment exists; R3: no next stepGift cards cannot be combined with other payments.

    Wrong: split payment exists. This one costs sales.

  2. 2"How do I repot a cactus?"

    A, passWear gloves or use folded newspaper, use cactus soil, and wait a week before watering.

    B, fails R3: thirteen sentencesHere are 12 tips for repotting a cactus. Tip 1: choose a pot with a hole in the bottom. (11 tips follow.)

    Correct, but far too long for a chat answer.

  3. 3"Where is my refund?"

    A, passRefunds take 3 to 5 working days; yours was sent on Monday.

    B, fails R3: no next stepYour refund has been sent.

    Correct, but it lost the one thing the customer asked: when the money arrives.

The wrong payment answer costs sales. Fix that case, rerun, and only then ship.

Show as text
VersionPassesPass rate
A41 of 5082%
B44 of 5088%

Case by case: 38 stayed a pass, 6 went from fail to pass, 3 went from pass to fail, 3 stayed a fail. 41 + 6 − 3 = 44.

  1. "Can I pay with a gift card and a bank card together?" A: Yes: at checkout, choose split payment. B: Gift cards cannot be combined with other payments. Wrong: split payment exists. This one costs sales.
  2. "How do I repot a cactus?" A: Wear gloves or use folded newspaper, use cactus soil, and wait a week before watering. B: Here are 12 tips for repotting a cactus. Tip 1: choose a pot with a hole in the bottom. (11 tips follow.) Correct, but far too long for a chat answer.
  3. "Where is my refund?" A: Refunds take 3 to 5 working days; yours was sent on Monday. B: Your refund has been sent. Correct, but it lost the one thing the customer asked: when the money arrives.
fig. 3 The same 50 cases, before and after a prompt change. B passes 3 more, but that is 6 fixed and 3 broken: the average hides the 3 that got worse (circled).

An average that goes up can hide answers that got worse. Read the regressions first.

04 · Graders

Check the checker.

With hundreds of cases, you let a model grade against the rules. It is fast and mostly right, but it makes its own mistakes. So once a month a person grades 20 of its verdicts again. Three mistakes I have seen in this kind of setup, and one case it gets right:

One answer

Customer"When does order 5120 arrive?"

AssistantYour order 5120 arrives on Thursday before 12:00.

The person also checksOrder data: 5120 is paid, not shipped yet, no delivery date.

Grader model

Passspecific and polite

reads the question, the answer and the rules

A person

Failthere is no delivery date yet

also checks the policy and the order data

Too lenient: the grader cannot see the order data, so an invented date looks fine. A bad answer goes out.

Spot check: 20 verdicts

17 of 20 agree (85%)

3 wrong: 2 too strict, 1 too lenient

  • agrees with the person
  • wrong
  • this case
Show as text
CaseGrader saidA person saidAgree?
"Is the monstera safe for cats?" No, monstera is toxic to cats. Put it out of reach, or choose a calathea, which is safe for pets.Pass: correct and shortPass: all three rules holdYes
"Can I give my cactus normal plant food?" I do not know for sure, and I would rather not guess. I have asked a colleague, who will email you today.Fail: not helpfulPass: honest, with a clear next stepNo
"When does order 5120 arrive?" Your order 5120 arrives on Thursday before 12:00.Pass: specific and politeFail: there is no delivery date yetNo
"Kan ik mijn bestelling nog aanpassen?" (Dutch: can I still change my order?) Ja, zolang hij nog niet verzonden is. Ik geef je vraag door aan het team; je hoort vandaag van ons.Fail: not in EnglishPass: the rules never say English onlyNo

Spot check of 20 verdicts: the grader agrees with the person on 17 and is wrong on 3 (2 too strict, 1 too lenient).

fig. 4 Check the checker. A person regrades 20 of the grader's verdicts: it agrees on 17 and is wrong on 3, twice too strict and once too lenient. The lenient one lets a bad answer out.

An automated grader is a second AI feature. It needs its own small test set.

Questions

How many test cases do I need to start?

Around 30 real cases is enough to catch most regressions in a first version. Add every bad answer you see in production.