The previous explainers showed similarity search working. This one shows its blind spot. It happens whenever you have many documents that look alike: subscription plans, contract versions, policies per country. This page uses three invented phone plans and prepared scores; nothing is sent to a model.
01 · The problem
The right text for the wrong customer.
Three plans with the same structure and nearly the same words. A customer asks "How much data can I use abroad?" Pick who is asking and watch what reaches the search, and what comes back.
Who asks
Sam
Plan: Plus
"How much data can I use abroad?"
The plan stays here: the question does not mention it.
The search, over every help text
Searched for "How much data can I use abroad?"
- 1 Data abroad: 1 GB per month in the EU, 2 euros per extra GB. Basic plan, page 3goes into the reply
- 2 Data abroad: 10 GB per month in the EU, 1 euro per extra GB. Plus plan, page 3Sam's plan
- 3 Travel tips: switch off automatic app updates when abroad. Help article, travel
- 4 Data abroad: unlimited in the EU, 5 GB worldwide. Max plan, page 3
The same 4 results in the same order, whoever asks. The dashed line marks the top score.
The top match is Basic, so Sam is told 1 GB per month in the EU. Sam's own plan, Plus, comes second with 0.90: 0.01 lower.
Show as text
| Place | Text the search found | Where it came from | Score for "How much data can I use abroad?" |
|---|---|---|---|
| 1 | Data abroad: 1 GB per month in the EU, 2 euros per extra GB. | Basic plan, page 3 | 0.91 |
| 2 | Data abroad: 10 GB per month in the EU, 1 euro per extra GB. | Plus plan, page 3 | 0.90 |
| 3 | Travel tips: switch off automatic app updates when abroad. | Help article, travel | 0.84 |
| 4 | Data abroad: unlimited in the EU, 5 GB worldwide. | Max plan, page 3 | 0.83 |
| Who asks | Their plan | Place of their plan | What happens |
|---|---|---|---|
| Sam | Plus | second | The top match is Basic, so Sam is told 1 GB per month in the EU. Sam's own plan, Plus, comes second with 0.90: 0.01 lower. |
| Lieke | Basic | first | The top match is Basic, and Lieke has Basic: right, but only because Basic happened to score highest. |
| Omar | Max | fourth | The top match is Basic, so Omar is told 1 GB per month in the EU. Omar's own plan, Max, comes fourth with 0.83: 0.08 lower, below a travel tip. |
With look-alike documents, the scores of right and wrong answers are almost equal. The search cannot tell who is asking.
02 · Why
Chunks lose track of who they belong to.
A search does not read whole documents. It reads chunks: the documents cut into pieces first (how cutting works is shown in fig. 4 of the embeddings page and fig. 3 of the vector search page). Every plan here is written the same way: its name on page 1, the rule for data abroad on page 3. Follow one plan into the index.
The Plus plan, as written
Plus plan
Calls: unlimited
1the plan name: page 1
Data abroad
10 GB per month in the EU, 1 euro per extra GB.
2the rule: page 3
In the index, next to the other plans
- Data abroad: 1 GB per month in the EU, 2 euros per extra GB. from the Basic plan (not in the text)
- Data abroad: 10 GB per month in the EU, 1 euro per extra GB. from the Plus plan (not in the text)
- Data abroad: unlimited in the EU, 5 GB worldwide. from the Max plan (not in the text)
3three rules with the same words: data, abroad, GB, EU
The question
"How much data can I use abroad?"
which plan?the question does not say
4no plan in the question, so nothing tells the three rules apart
Show as text
| Plan | Page 1 | Page 3 | The chunk in the index |
|---|---|---|---|
| Basic | Basic plan. Calls: 100 minutes. | Data abroad: 1 GB per month in the EU, 2 euros per extra GB. | Data abroad: 1 GB per month in the EU, 2 euros per extra GB. (no plan name in it) |
| Plus | Plus plan. Calls: unlimited. | Data abroad: 10 GB per month in the EU, 1 euro per extra GB. | Data abroad: 10 GB per month in the EU, 1 euro per extra GB. (no plan name in it) |
| Max | Max plan. Calls: unlimited. | Data abroad: Unlimited in the EU, 5 GB worldwide. | Data abroad: unlimited in the EU, 5 GB worldwide. (no plan name in it) |
- The plan name is on page 1.
- The rule for data abroad is on page 3, so its chunk does not name the plan.
- The three rules share the words data, abroad, GB, EU.
- The question, "How much data can I use abroad?", names no plan, so nothing in it tells the three rules apart.
You could stamp each chunk with its plan, as fig. 3 of the vector search page does, and filter on that label (fig. 4 there). That only helps once you know which plan to filter on. Here the plan belongs to the person asking, and the question does not carry it.
A chunk is a sentence without an owner unless you give it one, and even then the question does not say which owner to look for.
03 · The fix
Put the facts on the record they belong to.
Instead of searching all plan documents, the facts that differ per plan become fields on a plan record, and each customer record points to a plan. The chat already knows who is asking, because the person is signed in. Pick a customer and follow the lookup.
The chat
Signed in as Sam
"How much data can I use abroad?"
Customer records
NamePlan
SamPlus
LiekeBasic
OmarMax
Plan records
- Basic Data abroad1 GB per month in the EU Extra data2 euros per extra GB
- Plus Data abroad10 GB per month in the EU Extra data1 euro per extra GB
- Max Data abroadUnlimited in the EU, 5 GB worldwide Extra dataNot needed
The reply
On your Plus plan: 10 GB per month in the EU. Extra data: 1 euro per extra GB.
Show as text
| Signed in as | Customer record says | Plan record fields | Reply |
|---|---|---|---|
| Sam | plan = Plus | Data abroad: 10 GB per month in the EU. Extra data: 1 euro per extra GB. Calls: Unlimited. | On your Plus plan: 10 GB per month in the EU. Extra data: 1 euro per extra GB. |
| Lieke | plan = Basic | Data abroad: 1 GB per month in the EU. Extra data: 2 euros per extra GB. Calls: 100 minutes. | On your Basic plan: 1 GB per month in the EU. Extra data: 2 euros per extra GB. |
| Omar | plan = Max | Data abroad: Unlimited in the EU, 5 GB worldwide. Extra data: Not needed. Calls: Unlimited. | On your Max plan: unlimited in the EU, 5 GB worldwide. Extra data: not needed. |
Look up what belongs to the customer; search only for what is general.
04 · Side by side
Records for your own facts, search for the rest.
Not every question is about the customer's own plan. Some are the same for everyone, and those are what similarity search is good at. Pick a question: all three customers ask it, and it is answered both ways.
The question
"How much data can I use abroad?"
The answer depends on the plan.
Asked by Sam (Plus), Lieke (Basic) and Omar (Max).
- Sam has Plus Search 1 GB per month in the EU the Basic rule, but Sam has Plus Record 10 GB per month in the EU from Sam's record: Plus
- Lieke has Basic Search 1 GB per month in the EU the Basic rule, and Lieke has Basic Record 1 GB per month in the EU from Lieke's record: Basic
- Omar has Max Search 1 GB per month in the EU the Basic rule, but Omar has Max Record Unlimited in the EU, 5 GB worldwide from Omar's record: Max
Fits the customer Search 1 of 3 Record 3 of 3
Search gives all three customers the Basic rule: right for Lieke, by luck. The records give each customer the facts of their own plan.
Show as text
"Any tips to save data when I travel?" (the answer is the same for everyone)
| Customer | Similarity search | Record lookup |
|---|---|---|
| Sam (Plus) | Switch off automatic app updates when abroad. (travel article, the same for everyone; fits) | No field for this, so it goes to search |
| Lieke (Basic) | Switch off automatic app updates when abroad. (travel article, the same for everyone; fits) | No field for this, so it goes to search |
| Omar (Max) | Switch off automatic app updates when abroad. (travel article, the same for everyone; fits) | No field for this, so it goes to search |
A travel tip is the same for everyone, so one search result fits all three customers. The plan records have no field for it, so this question goes to search. Search fits 3 of 3; the records have nothing for this question.
"How much data can I use abroad?" (the answer depends on the plan)
| Customer | Similarity search | Record lookup |
|---|---|---|
| Sam (Plus) | 1 GB per month in the EU (the Basic rule, but Sam has Plus; does not fit) | 10 GB per month in the EU (from Sam's record: Plus; fits) |
| Lieke (Basic) | 1 GB per month in the EU (the Basic rule, and Lieke has Basic; fits) | 1 GB per month in the EU (from Lieke's record: Basic; fits) |
| Omar (Max) | 1 GB per month in the EU (the Basic rule, but Omar has Max; does not fit) | Unlimited in the EU, 5 GB worldwide (from Omar's record: Max; fits) |
Search gives all three customers the Basic rule: right for Lieke, by luck. The records give each customer the facts of their own plan. Search fits 1 of 3; the records fit 3 of 3.
Records give each customer the one right answer. Search gives everyone the most similar text: fine for a general tip, wrong for a plan.