The most common way to test an AI feature is trying five questions and going with a feeling. It works until the next change quietly breaks something. An eval is the product manager's version of a test: a fixed set of questions, written rules, and a comparison. Everything here is invented.
01 · Your gut
Grade six answers yourself.
A support assistant for a plant webshop gave these answers. Mark each as pass or fail. My grade turns over as you go, and the tally circles every answer we grade differently.
-
1Customer"My order has not arrived, it has been 5 days."
AssistantSorry about the wait. Your order shipped on Tuesday and tracking shows delivery tomorrow before 18:00; here is the tracking link.
My grade: pass. Uses the order data and gives a next step.
-
2Customer"Can I return a plant that died?"
AssistantPlants are living products, so unfortunately we cannot accept returns.
My grade: fail. Wrong: the shop has a 30-day plant guarantee, so the customer loses money they are owed. And it leaves them with no next step.
-
3Customer"Do you ship to Belgium?"
AssistantGreat question! We love our Belgian customers and we are always happy to help. Shipping to Belgium is possible. Many customers there order every month. Let us know if you have other questions!
My grade: fail. Friendly, but five sentences and no price, delivery time or next step. Fails the length rule.
-
4Customer"Is the monstera safe for cats?"
AssistantNo, monstera is toxic to cats. Put it out of reach, or choose a calathea, which is safe for pets.
My grade: pass. Correct, short, and offers an alternative.
-
5Customer"Cancel my order 4411."
AssistantDone, order 4411 is cancelled.
My grade: fail. The assistant cannot cancel orders, so the order is still open; it should hand over to the team. A broken promise, and no word on the refund.
-
6Customer"Your app is terrible."
AssistantSorry to hear that. What went wrong? I will pass it on to the app team today.
My grade: pass. Short, kind, asks for details, realistic promise.
Pass or fail, per answer
You: 0 passMe: 3 of 6 pass
- pass
- fail
- mine, until you grade it
I pass 3 of 6 (50%). With JavaScript on, you grade them too.
Show as text
| # | Customer asks | Assistant answers | My grade | Why |
|---|---|---|---|---|
| 1 | My order has not arrived, it has been 5 days. | Sorry about the wait. Your order shipped on Tuesday and tracking shows delivery tomorrow before 18:00; here is the tracking link. | Pass | Uses the order data and gives a next step. |
| 2 | Can I return a plant that died? | Plants are living products, so unfortunately we cannot accept returns. | Fail | Wrong: the shop has a 30-day plant guarantee, so the customer loses money they are owed. And it leaves them with no next step. |
| 3 | Do you ship to Belgium? | Great question! We love our Belgian customers and we are always happy to help. Shipping to Belgium is possible. Many customers there order every month. Let us know if you have other questions! | Fail | Friendly, but five sentences and no price, delivery time or next step. Fails the length rule. |
| 4 | Is the monstera safe for cats? | No, monstera is toxic to cats. Put it out of reach, or choose a calathea, which is safe for pets. | Pass | Correct, short, and offers an alternative. |
| 5 | Cancel my order 4411. | Done, order 4411 is cancelled. | Fail | The assistant cannot cancel orders, so the order is still open; it should hand over to the team. A broken promise, and no word on the refund. |
| 6 | Your app is terrible. | Sorry to hear that. What went wrong? I will pass it on to the app team today. | Pass | Short, kind, asks for details, realistic promise. |
I pass 3 of 6.
Your gut is a test set of one person on one day. Useful to start, not enough to decide.
02 · The rubric
Write down what a good answer must do.
Most disagreements come from rules nobody wrote down. Is a friendly but long answer a pass? Here are three rules. Pick an answer and watch it go past them: one no is enough to fail.
The answer
3Customer"Do you ship to Belgium?"
AssistantGreat question! We love our Belgian customers and we are always happy to help. Shipping to Belgium is possible. Many customers there order every month. Let us know if you have other questions!
Three written rules
- R1 Correct according to our policy and the order data. yes
- R2 Promises nothing the assistant cannot do. yes
- R3 At most three sentences, and a clear next step. no: five sentences, and no next step
The result
FailOne no is enough.
All six against the rules
| Answer | R1 | R2 | R3 | Rules | You |
|---|---|---|---|---|---|
| 1 Late order | yes | yes | yes | pass | ? |
| 2 Dead plant | no | yes | no | fail | ? |
| 3 Belgium | yes | yes | no | fail | ? |
| 4 Monstera | yes | yes | yes | pass | ? |
| 5 Cancel order | no | no | no | fail | ? |
| 6 Bad app | yes | yes | yes | pass | ? |
| Yes, of 6 | 4 | 5 | 3 | 3 pass |
You: your grades from fig. 1. Grade them there to compare.
Show the rules and all six results as text
- R1: Correct according to our policy and the order data.
- R2: Promises nothing the assistant cannot do.
- R3: At most three sentences, and a clear next step.
A next step tells the customer what happens next or what they can do now.
| Answer | R1 | R2 | R3 | Result |
|---|---|---|---|---|
| 1. My order has not arrived, it has been 5 days. | yes | yes | yes | Pass |
| 2. Can I return a plant that died? | no: the shop has a 30-day plant guarantee | yes | no: no next step | Fail |
| 3. Do you ship to Belgium? | yes | yes | no: five sentences, and no next step | Fail |
| 4. Is the monstera safe for cats? | yes | yes | yes | Pass |
| 5. Cancel my order 4411. | no: the order data says it is still open | no: it cannot cancel orders: the team does that | no: no next step | Fail |
| 6. Your app is terrible. | yes | yes | yes | Pass |
Three written rules turn a feeling into something two people, or a machine, can apply the same way.
03 · Versions
Compare versions, then read what got worse.
You change the prompt. On the 50-case set, version B passes more cases than version A. Ship it? Look at the cases that went from pass to fail first: the same case sits in the same spot in both grids.
Version A
41 of 50 pass (82%)
Version B
44 of 50 pass (88%)
The averages
B looks better: 3 more cases pass, 6 points up.
Case by case
6 fixed
3 broke
41 + 6 fixed − 3 broke = 44. Also 38 stayed a pass and 3 stayed a fail.
- pass
- fail
- fixed in B
- broke in B
The averages do not say which cases changed. Switch to case by case.
The 3 that got worse, to read before you ship
-
1"Can I pay with a gift card and a bank card together?"
A, passYes: at checkout, choose split payment.
B, fails R1: split payment exists; R3: no next stepGift cards cannot be combined with other payments.
Wrong: split payment exists. This one costs sales.
-
2"How do I repot a cactus?"
A, passWear gloves or use folded newspaper, use cactus soil, and wait a week before watering.
B, fails R3: thirteen sentencesHere are 12 tips for repotting a cactus. Tip 1: choose a pot with a hole in the bottom. (11 tips follow.)
Correct, but far too long for a chat answer.
-
3"Where is my refund?"
A, passRefunds take 3 to 5 working days; yours was sent on Monday.
B, fails R3: no next stepYour refund has been sent.
Correct, but it lost the one thing the customer asked: when the money arrives.
The wrong payment answer costs sales. Fix that case, rerun, and only then ship.
Show as text
| Version | Passes | Pass rate |
|---|---|---|
| A | 41 of 50 | 82% |
| B | 44 of 50 | 88% |
Case by case: 38 stayed a pass, 6 went from fail to pass, 3 went from pass to fail, 3 stayed a fail. 41 + 6 − 3 = 44.
- "Can I pay with a gift card and a bank card together?" A: Yes: at checkout, choose split payment. B: Gift cards cannot be combined with other payments. Wrong: split payment exists. This one costs sales.
- "How do I repot a cactus?" A: Wear gloves or use folded newspaper, use cactus soil, and wait a week before watering. B: Here are 12 tips for repotting a cactus. Tip 1: choose a pot with a hole in the bottom. (11 tips follow.) Correct, but far too long for a chat answer.
- "Where is my refund?" A: Refunds take 3 to 5 working days; yours was sent on Monday. B: Your refund has been sent. Correct, but it lost the one thing the customer asked: when the money arrives.
An average that goes up can hide answers that got worse. Read the regressions first.
04 · Graders
Check the checker.
With hundreds of cases, you let a model grade against the rules. It is fast and mostly right, but it makes its own mistakes. So once a month a person grades 20 of its verdicts again. Three mistakes I have seen in this kind of setup, and one case it gets right:
One answer
Customer"When does order 5120 arrive?"
AssistantYour order 5120 arrives on Thursday before 12:00.
The person also checksOrder data: 5120 is paid, not shipped yet, no delivery date.
Grader model
Passspecific and polite
reads the question, the answer and the rules
A person
Failthere is no delivery date yet
also checks the policy and the order data
Too lenient: the grader cannot see the order data, so an invented date looks fine. A bad answer goes out.
Spot check: 20 verdicts
17 of 20 agree (85%)
3 wrong: 2 too strict, 1 too lenient
- agrees with the person
- wrong
- this case
Show as text
| Case | Grader said | A person said | Agree? |
|---|---|---|---|
| "Is the monstera safe for cats?" No, monstera is toxic to cats. Put it out of reach, or choose a calathea, which is safe for pets. | Pass: correct and short | Pass: all three rules hold | Yes |
| "Can I give my cactus normal plant food?" I do not know for sure, and I would rather not guess. I have asked a colleague, who will email you today. | Fail: not helpful | Pass: honest, with a clear next step | No |
| "When does order 5120 arrive?" Your order 5120 arrives on Thursday before 12:00. | Pass: specific and polite | Fail: there is no delivery date yet | No |
| "Kan ik mijn bestelling nog aanpassen?" (Dutch: can I still change my order?) Ja, zolang hij nog niet verzonden is. Ik geef je vraag door aan het team; je hoort vandaag van ons. | Fail: not in English | Pass: the rules never say English only | No |
Spot check of 20 verdicts: the grader agrees with the person on 17 and is wrong on 3 (2 too strict, 1 too lenient).
An automated grader is a second AI feature. It needs its own small test set.