Lab / Decision tool / 4 questions

No. 07Decision tool5 min

Do you need a vector database?

A four-question decision tool: put everything in the prompt, compress it first, search pieces, or look up records.

Short version

Content that fits in one window goes straight into the prompt. Big content needs search. Many look-alike documents need structure. Start simple and measure before you add a database.

"We need RAG" is often the first sentence of an AI feature plan, and often the wrong first step. Models now read hundreds of pages in one go. This page helps you pick the simplest option that works. Sizes and costs are rough rules of thumb, last checked 2026-11-13, and every document on it is invented.

01 · Measure

Start by measuring how much content you have.

The context window is how much text a model can read at once. Assume 200,000 tokens, roughly 400 pages. Drag to see where your content lands. The window is drawn twice: up close, where a dot is about a page, and from afar, where a square is a whole window.

Your content

200 pages

about 100,000 tokens

1 page is about 500 tokens: 1 dot

One context window, up close

200 of 400 dots

All of it, from afar

  • 1 square = 1 whole window, the grid up close
  • your content: 0.5 windows

Fits: uses 50% of the window.

Show as text
ContentTokensIn one windowWhole windowsVerdict
20 pages10,00020 of 400 dots0.05Fits: uses 5% of the window.
200 pages100,000200 of 400 dots0.5Fits: uses 50% of the window.
2,000 pages1,000,000400 of 400 dots5Does not fit: 5 times the window. 1,600 of the 2,000 pages are left out.
20,000 pages10,000,000400 of 400 dots50Does not fit: 50 times the window. 19,600 of the 20,000 pages are left out.
fig. 1 Measure before you build. Up close, 1 dot is 500 tokens, about a page, and the window holds 400. From afar, 1 square is a whole window: up to 400 pages fits as it is; 2,000 pages is five windows.

Under a few hundred pages, you may not need search at all.

02 · Options

There are four ways to give a model your knowledge.

Small content goes straight into the prompt. Bigger content needs something in between that picks what the model reads: a summary made in advance, a search that finds the right pieces, or records looked up by who is asking. Follow one invented question through each option: what the model has to read for it, and what you have to build.

This page only decides between them. How a search index is built and read is drawn in fig. 1 of How vector search works. Why look-alike documents need records, and how the lookup works, is drawn in fig. 3 of When look-alike documents fool the search.

A customer asks

"Is my e-bike covered if it is stolen abroad?"

300 pages of policy terms for three bike plans, plus help articles: 150,000 tokens

What you build in between

Nothing in between

  • Your software pastes every page in with the question.
  • Prompt caching makes repeats cheap.

What the model reads, to scale

150,000 tokens: all 300 pages, with every question

Use when
Small content that fits in one window.
Main risk
Content outgrows the window.

1 dot = 500 tokens, as in fig. 1; the pale dots are the rest of the window.

Show as text
OptionUse whenWhat you buildCostMain riskThe model reads, per question
Put it all in the promptSmall content that fits in one window.Nothing in between: Your software pastes every page in with the question. Prompt caching makes repeats cheap.Lowest build effort. Cost per question grows with size; prompt caching makes repeats cheap.Content outgrows the window.All 300 pages, with every question: 150,000 tokens
Compress first, then promptToo big for the prompt as is, but mostly repetition and boilerplate.A summary step: Summarise each document once. Rerun it for a document that changes.A one-off summarising step per document, rerun when it changes.The summary drops a detail someone needs. Test against the full text.A 40-page summary of all 300: 20,000 tokens
Search pieces (RAG)Content too big for one window that changes often, or has distinct topics.A search index: Cut every document into pieces and index them. Keep the index up to date. Find the 5 closest pieces for each question.An index to build and keep up to date; more moving parts.Retrieval misses the right piece; needs testing with real questions.The 5 closest pieces, about a page each: 2,500 tokens
Look up recordsMany look-alike documents where the right one depends on who is asking.Records with fields: Put the facts that differ into fields, one record per plan. Look up the record of whoever is asking. Give every record an owner.Facts need to live in fields on records; work up front.Records go stale if nobody owns them.The asker's plan record: 300 tokens
fig. 2 The same 300 pages and the same question, four ways. Each option to the right sends less to read per question (the pen dots, to scale); every option after the first needs something in between that you build and keep running. Sizes are illustrative.

Each step up adds machinery to build and keep running. Climb only when you must.

03 · Decide

Four questions, one recommendation.

Answer four questions about your own content. The checks are asked in order, and the first yes decides; with three no's, it is search. How bad a wrong answer is never changes the option: it adds checks on top. Change any answer to follow the path again.

Your situation

How much content?

200 pages

How often does it change?

Rarely

Many near-identical documents (per customer, plan, version)?

No

How bad is a wrong answer?

Annoying

Asked in orderThe option

  1. 1Many look-alike documents?no

    Look up recordsRecords with fields

  2. 2Fits in one window?yes

    Put it all in the promptNothing in between

  3. 3Up to 4,000 pages, and changes rarely?

    Compress first, then promptA summary step

  4. No to all three

    Search pieces (RAG)A search index

Then: is a wrong answer costly?no

  • Ask for citations
  • Keep a test set of real questions
  • Put a person on the risky answers

200 pages fits in one context window. Put it in the prompt, use prompt caching, and measure.

Show the decision rules as text
IfThen
Many look-alike documentsLook up records (search only for general questions)
Fits in the window (up to about 400 pages), however often it changesPut it all in the prompt (send the fresh version with every question)
Too big for one window, up to about 4,000 pages, and changes rarelyCompress first, then prompt
Too big for one window, and changes weekly or faster or is over about 4,000 pagesSearch pieces (hybrid search)
A wrong answer is costlyAdd citations, a test set and a human check on top of any option
ContentChangesNo look-alikesMany look-alikes
20 pagesRarelyPut it all in the promptLook up records
20 pagesWeeklyPut it all in the promptLook up records
20 pagesDaily or fasterPut it all in the promptLook up records
200 pagesRarelyPut it all in the promptLook up records
200 pagesWeeklyPut it all in the promptLook up records
200 pagesDaily or fasterPut it all in the promptLook up records
2,000 pagesRarelyCompress first, then promptLook up records
2,000 pagesWeeklySearch pieces (RAG)Look up records
2,000 pagesDaily or fasterSearch pieces (RAG)Look up records
20,000 pagesRarelySearch pieces (RAG)Look up records
20,000 pagesWeeklySearch pieces (RAG)Look up records
20,000 pagesDaily or fasterSearch pieces (RAG)Look up records
fig. 3 The first yes decides; with three no's, it is search. A costly wrong answer never changes the option: it adds checks on top of it.

The answer changes most with size and with look-alikes, less with anything else.