Vector databases and RAG: how AI finds the right paragraph in thousands of documents.
When you ask AI about a contract or a price list, it doesn't read the whole archive. It finds a few paragraphs that belong to the question and answers from them. This is called RAG, and at its heart is a vector database. We explain how it works, what works better today than a year ago and what to watch out for.
Why AI doesn't read all your documents at once.
The first idea is to give AI all your documents and let it figure things out. With dozens of pages, that works. With thousands, it doesn't. The longer the text a model gets, the easier it is to miss the important paragraph. And every question over the whole archive is slow and expensive. That is why documents aren't sent to AI whole. First, the parts that belong to the question are found.
“A company has 4,000 pages of contracts, policies and emails. A salesperson asks what the notice period is in the contract with Novák. AI doesn't read 4,000 pages. The search returns three paragraphs, AI answers from them and adds a link to the page of the contract.”
RAG (retrieval-augmented generation, an answer supplemented by search) has two steps: first find, then answer. The quality of the answer therefore depends mainly on the first step. If the search returns the wrong paragraphs, even the best model won't answer correctly.
A vector is meaning written in numbers.
Classic search looks for the same words. Vector search looks for the same meaning. Each chunk of text passes through a so-called embedding model, which turns it into a vector: a series of hundreds to thousands of numbers. Texts with similar meanings get similar numbers, even if they don't share a single word.
With a person
A librarian doesn't search by exact title. When you say “something about complaints in an e-shop”, she knows which shelf is related, even if that word isn't on the book's spine.
With AI
The question “can I return broken goods?” finds a paragraph about a “defective product and withdrawal from the contract”. Different words, same meaning.
Similarity is measured as the distance between vectors, most often with cosine similarity. So the database doesn't have to compare the question with millions of chunks one by one, it uses approximate nearest neighbor search. The result is ready in a fraction of a second. But it is “most similar”, not “correct”. That is why further steps are needed.
How a RAG system works in six steps
The whole system can be broken into six steps. The first three are done once, when you add documents. The other three happen with every question.
| STEP | WHAT HAPPENS | WHEN |
|---|---|---|
| 1. Split the documents | The text is split into chunks, for example by paragraph | On upload |
| 2. Create vectors | Each chunk passes through an embedding model | On upload |
| 3. Store | Vectors are stored with their source, date and access rights | On upload |
| 4. Search | The question is turned into a vector and the nearest chunks are found | Per question |
| 5. Rank | Another model ranks the chunks by whether they actually answer | Per question |
| 6. Answer | AI answers only from the chunks found and cites the source | Per question |
Most errors happen in steps 1 and 4. Chunks that are too large are imprecise, chunks that are too small lose context. And search without filters for validity and access rights also returns an old price list or a document that person shouldn't see.
What works better today than plain vector search
The first RAG systems relied on vectors alone. Today they are combined with other techniques:
- Hybrid search: vectors find the meaning, keyword search finds exact terms such as a contract number or a product code.
- Reranking: another model reads the chunks found and ranks them by how well they actually answer the question.
- Context for chunks: each chunk gets a short description of where it comes from. Together with hybrid search and reranking, this cut failed retrievals by 67% in Anthropic's test.
- Relationship graphs and agentic search: for questions across many documents, a map of relationships between concepts helps (GraphRAG). An agent decides itself what and where to search, and if the results aren't enough, it searches again.
With a person
An experienced assistant doesn't look in just one drawer. She checks the index, leafs through the binder and, if she isn't sure, asks a colleague.
With AI
A good RAG system does the same: it searches in several ways, compares the results and searches again if they aren't enough.
When you don't need a vector database
If you have a few dozen pages that fit into one project in Claude, a vector database is unnecessary. AI reads them whole. If people ask about numbers, such as revenue or order status, those belong in a classic database. A vector database makes sense when there are many documents, they change, and people ask about them in their own words.
What RAG means for your company
RAG is a way for AI to answer from your documents instead of from memory. The vector database in it searches by meaning, not by words. It works well when the documents are in order, search runs in several ways and AI cites the source.
For a company, this leads to three decisions: which documents belong in the library and who keeps them up to date, who may see what, and where the data will be stored. The technology is widely available today, for example as the pgvector extension for PostgreSQL. The content is what decides.
Sources: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020); Anthropic, Introducing Contextual Retrieval (2024); IBM, What is a RAG vector database; Microsoft Research, GraphRAG (2024).
Ask yourself
If AI were asked about your internal rules today, would it find the current version or last year's?
