Lab / Explainer / 4 steps

No. 05Explainer6 min

When look-alike documents fool the search

Why similarity search mixes up near-identical documents, and how storing facts on the record they belong to fixes it.

Short version

When you have many documents that look alike (plans, contracts, versions), similarity search can return the right text for the wrong customer. Store the facts on the record they belong to and look them up by who is asking.

The previous explainers showed similarity search working. This one shows its blind spot. It happens whenever you have many documents that look alike: subscription plans, contract versions, policies per country. This page uses three invented phone plans and prepared scores; nothing is sent to a model.

01 · The problem

The right text for the wrong customer.

Three plans with the same structure and nearly the same words. A customer asks "How much data can I use abroad?" Pick who is asking and watch what reaches the search, and what comes back.

Who asks

Sam

Plan: Plus

"How much data can I use abroad?"

The plan stays here: the question does not mention it.

The search, over every help text

Searched for "How much data can I use abroad?"

  1. 1 Data abroad: 1 GB per month in the EU, 2 euros per extra GB. Basic plan, page 3goes into the reply 0.91
  2. 2 Data abroad: 10 GB per month in the EU, 1 euro per extra GB. Plus plan, page 3Sam's plan 0.90
  3. 3 Travel tips: switch off automatic app updates when abroad. Help article, travel 0.84
  4. 4 Data abroad: unlimited in the EU, 5 GB worldwide. Max plan, page 3 0.83

The same 4 results in the same order, whoever asks. The dashed line marks the top score.

The top match is Basic, so Sam is told 1 GB per month in the EU. Sam's own plan, Plus, comes second with 0.90: 0.01 lower.

Show as text
PlaceText the search foundWhere it came fromScore for "How much data can I use abroad?"
1Data abroad: 1 GB per month in the EU, 2 euros per extra GB.Basic plan, page 30.91
2Data abroad: 10 GB per month in the EU, 1 euro per extra GB.Plus plan, page 30.90
3Travel tips: switch off automatic app updates when abroad.Help article, travel0.84
4Data abroad: unlimited in the EU, 5 GB worldwide.Max plan, page 30.83
Who asksTheir planPlace of their planWhat happens
SamPlussecondThe top match is Basic, so Sam is told 1 GB per month in the EU. Sam's own plan, Plus, comes second with 0.90: 0.01 lower.
LiekeBasicfirstThe top match is Basic, and Lieke has Basic: right, but only because Basic happened to score highest.
OmarMaxfourthThe top match is Basic, so Omar is told 1 GB per month in the EU. Omar's own plan, Max, comes fourth with 0.83: 0.08 lower, below a travel tip.
fig. 1 The search only gets the question, so every customer gets the same ranking. Basic wins by a hundredth, and Sam, who has Plus, is told the Basic rule.

With look-alike documents, the scores of right and wrong answers are almost equal. The search cannot tell who is asking.

02 · Why

Chunks lose track of who they belong to.

A search does not read whole documents. It reads chunks: the documents cut into pieces first (how cutting works is shown in fig. 4 of the embeddings page and fig. 3 of the vector search page). Every plan here is written the same way: its name on page 1, the rule for data abroad on page 3. Follow one plan into the index.

The Plus plan, as written

page 1

Plus plan

Calls: unlimited

1the plan name: page 1

page 3

Data abroad

10 GB per month in the EU, 1 euro per extra GB.

2the rule: page 3

In the index, next to the other plans

  1. Data abroad: 1 GB per month in the EU, 2 euros per extra GB. from the Basic plan (not in the text)
  2. Data abroad: 10 GB per month in the EU, 1 euro per extra GB. from the Plus plan (not in the text)
  3. Data abroad: unlimited in the EU, 5 GB worldwide. from the Max plan (not in the text)

3three rules with the same words: data, abroad, GB, EU

The question

"How much data can I use abroad?"

which plan?the question does not say

4no plan in the question, so nothing tells the three rules apart

Show as text
PlanPage 1Page 3The chunk in the index
BasicBasic plan. Calls: 100 minutes.Data abroad: 1 GB per month in the EU, 2 euros per extra GB.Data abroad: 1 GB per month in the EU, 2 euros per extra GB. (no plan name in it)
PlusPlus plan. Calls: unlimited.Data abroad: 10 GB per month in the EU, 1 euro per extra GB.Data abroad: 10 GB per month in the EU, 1 euro per extra GB. (no plan name in it)
MaxMax plan. Calls: unlimited.Data abroad: Unlimited in the EU, 5 GB worldwide.Data abroad: unlimited in the EU, 5 GB worldwide. (no plan name in it)
  1. The plan name is on page 1.
  2. The rule for data abroad is on page 3, so its chunk does not name the plan.
  3. The three rules share the words data, abroad, GB, EU.
  4. The question, "How much data can I use abroad?", names no plan, so nothing in it tells the three rules apart.
fig. 2 The plan name is on page 1 and the rule on page 3, so the chunk with the rule does not say whose it is. The three rules share the same words, and the question does not say which plan it means.

You could stamp each chunk with its plan, as fig. 3 of the vector search page does, and filter on that label (fig. 4 there). That only helps once you know which plan to filter on. Here the plan belongs to the person asking, and the question does not carry it.

A chunk is a sentence without an owner unless you give it one, and even then the question does not say which owner to look for.

03 · The fix

Put the facts on the record they belong to.

Instead of searching all plan documents, the facts that differ per plan become fields on a plan record, and each customer record points to a plan. The chat already knows who is asking, because the person is signed in. Pick a customer and follow the lookup.

The chat

Signed in as Sam

"How much data can I use abroad?"

Customer records

NamePlan

SamPlus

LiekeBasic

OmarMax

Plan records

  1. Basic Data abroad1 GB per month in the EU Extra data2 euros per extra GB
  2. Plus Data abroad10 GB per month in the EU Extra data1 euro per extra GB
  3. Max Data abroadUnlimited in the EU, 5 GB worldwide Extra dataNot needed

The reply

On your Plus plan: 10 GB per month in the EU. Extra data: 1 euro per extra GB.

Show as text
Signed in asCustomer record saysPlan record fieldsReply
Samplan = PlusData abroad: 10 GB per month in the EU. Extra data: 1 euro per extra GB. Calls: Unlimited.On your Plus plan: 10 GB per month in the EU. Extra data: 1 euro per extra GB.
Liekeplan = BasicData abroad: 1 GB per month in the EU. Extra data: 2 euros per extra GB. Calls: 100 minutes.On your Basic plan: 1 GB per month in the EU. Extra data: 2 euros per extra GB.
Omarplan = MaxData abroad: Unlimited in the EU, 5 GB worldwide. Extra data: Not needed. Calls: Unlimited.On your Max plan: unlimited in the EU, 5 GB worldwide. Extra data: not needed.
fig. 3 Who is asking comes from the sign-in, not from the question. The software finds that customer's record, follows it to one plan, and copies that plan's fields into the reply.

Look up what belongs to the customer; search only for what is general.

04 · Side by side

Records for your own facts, search for the rest.

Not every question is about the customer's own plan. Some are the same for everyone, and those are what similarity search is good at. Pick a question: all three customers ask it, and it is answered both ways.

The question

"How much data can I use abroad?"

The answer depends on the plan.

Asked by Sam (Plus), Lieke (Basic) and Omar (Max).

  1. Sam has Plus Search 1 GB per month in the EU the Basic rule, but Sam has Plus Record 10 GB per month in the EU from Sam's record: Plus
  2. Lieke has Basic Search 1 GB per month in the EU the Basic rule, and Lieke has Basic Record 1 GB per month in the EU from Lieke's record: Basic
  3. Omar has Max Search 1 GB per month in the EU the Basic rule, but Omar has Max Record Unlimited in the EU, 5 GB worldwide from Omar's record: Max

Fits the customer Search 1 of 3 Record 3 of 3

Search gives all three customers the Basic rule: right for Lieke, by luck. The records give each customer the facts of their own plan.

Show as text

"Any tips to save data when I travel?" (the answer is the same for everyone)

CustomerSimilarity searchRecord lookup
Sam (Plus)Switch off automatic app updates when abroad. (travel article, the same for everyone; fits)No field for this, so it goes to search
Lieke (Basic)Switch off automatic app updates when abroad. (travel article, the same for everyone; fits)No field for this, so it goes to search
Omar (Max)Switch off automatic app updates when abroad. (travel article, the same for everyone; fits)No field for this, so it goes to search

A travel tip is the same for everyone, so one search result fits all three customers. The plan records have no field for it, so this question goes to search. Search fits 3 of 3; the records have nothing for this question.

"How much data can I use abroad?" (the answer depends on the plan)

CustomerSimilarity searchRecord lookup
Sam (Plus)1 GB per month in the EU (the Basic rule, but Sam has Plus; does not fit)10 GB per month in the EU (from Sam's record: Plus; fits)
Lieke (Basic)1 GB per month in the EU (the Basic rule, and Lieke has Basic; fits)1 GB per month in the EU (from Lieke's record: Basic; fits)
Omar (Max)1 GB per month in the EU (the Basic rule, but Omar has Max; does not fit)Unlimited in the EU, 5 GB worldwide (from Omar's record: Max; fits)

Search gives all three customers the Basic rule: right for Lieke, by luck. The records give each customer the facts of their own plan. Search fits 1 of 3; the records fit 3 of 3.

fig. 4 For a question about the customer's own plan, search hands everyone the same text and fits one customer by luck, while the records fit all three. For a question that is the same for everyone, search is the right tool and the records have nothing to add.

Records give each customer the one right answer. Search gives everyone the most similar text: fine for a general tip, wrong for a plan.