When we learn about RAG, the usual flow feels very natural. A user asks a question, we create an embedding, search the vector database, retrieve the closest chunks, and send them to the LLM. After seeing this pattern a few times, it is easy to assume that every RAG question should go through similarity search.
But that is not always necessary.
Sometimes the application already knows exactly where the answer should come from. In those cases, asking the vector database to discover the right content again can add work without adding much value.
A Simple Example
Imagine an application that stores user manuals for many household products. It may contain manuals for washing machines, televisions, coffee machines, air conditioners, and many other products. A user opens the manual for a particular washing machine and asks
How do I clean the filter?
If we only had the question and thousands of documents, similarity search would make sense. We would need to discover which pieces of information are most relevant. But in this situation, the application already knows something important.
The user is currently reading one specific washing machine manual. We already know where to look.
Selected Washing Machine Manual
↓
"How do I clean the filter?"
↓
Relevant content from this manual
↓
LLM Answer
This changes the retrieval problem quite a lot.
What Similarity Search Is Really Helping Us Do
Similarity search is useful when we have a large amount of information and we do not know which part contains the answer. Imagine the user starts from a general support chat and asks
Why is my washing machine not draining?
Now the system may have thousands of chunks from many manuals and troubleshooting guides. Similarity search can help us find content related to ideas such as drainage problems, blocked filters, drain hoses, and error codes even when the exact words in the documents are different.
User Question
↓
Query Embedding
↓
Large Knowledge Base
↓
Similarity Search
↓
Most Relevant Chunks
Here, similarity search is solving a real problem. It is helping us discover where the useful information is. But if the application already knows the document, the situation is different.
When We Already Know the Scope
Suppose the washing machine manual has already been divided into ten chunks.The user is inside that manual and asks a question. One possible approach is still to create an embedding and ask the vector database for the top three chunks.
Known Manual
↓
10 Chunks
↓
Similarity Search
↓
Top 3 Chunks
This can work. But we should ask ourselves whether similarity search is actually helping. If the document is small enough, we may simply retrieve all ten chunks and provide the useful content to the next step.
Known Manual
↓
10 Chunks
↓
Retrieve the Content
↓
LLM
We already know those chunks belong to the document the user is asking about. There may be little value in asking a similarity algorithm to choose only three of them. In some cases, choosing only a few chunks can even remove context that would have been useful.
Imagine the Answer Is Spread Across Several Sections
Suppose the manual explains filter cleaning in three different places. One section explains where the filter is located. Another explains how to remove it. A later section explains how often it should be cleaned.
If we ask for only the top two most similar chunks, we may get the location and removal instructions but miss the maintenance information. The retrieval was technically correct. It returned the chunks most similar to the question.
But similarity does not always mean completeness. If the document is small and already known, reading all of the relevant scope may give the LLM a more complete picture.
That is why retrieval should depend on what we actually know about the request.
This Does Not Mean We Should Always Retrieve Everything
There is an important limit here. Imagine the selected manual is 400 pages long. Sending the whole document to the LLM would obviously not be a good idea. It would increase the amount of context, increase cost, and make it harder to focus on the useful information.
In that situation, similarity search becomes useful again. The difference is that we do not need to search every document in the knowledge base. We can search only inside the manual we already know is relevant.
Selected Manual
↓
Hundreds of Chunks
↓
Similarity Search Within That Manual
↓
Relevant Chunks
↓
LLM
So the decision is not simply between using vector search and never using it. The better question is whether we still need help discovering the right content.
A Simple Way to Decide
I like to think about retrieval in three situations.
If we have a large knowledge base and do not know where the answer is, similarity search is very useful.
For Unknown Scope
Thousands of Chunks
↓
Similarity Search
If we already know the document and it contains only a small amount of content, retrieving the whole useful scope can be simpler.
For Known Scope
Small Amount of Content
↓
Direct Retrieval
If we know the document but it is still very large, we can combine both ideas.
Known Scope
Large Amount of Content
↓
Similarity Search Inside That Scope
This is often a much better way to think about RAG than automatically sending every question through the same retrieval pipeline.
Top K Is Not Always the Right Answer
Many RAG examples use something like this.
topK: 5
That means the system returns the five closest vectors. There is nothing wrong with that, but 5 is not a magic number.
Imagine the information needed to answer a question is spread across seven relevant chunks. Returning only five may lose something important. On the other hand, returning twenty chunks when only two matter can fill the prompt with unnecessary information.
This is another reason why knowing the scope helps. If the application already knows there are only six chunks in the selected document, asking similarity search for the top five may not be useful at all. We may simply use all six. The retrieval strategy should fit the situation instead of following the same number for every question.
Retrieval Does Not Have to Be One Fixed Pipeline
It is easy to imagine RAG as one permanent flow.
Question
↓
Embedding
↓
Vector Search
↓
Top K
↓
LLM
Real applications can be more flexible. The retrieval path can change depending on what is already known.
User Question
↓
What scope do we already know?
↓
Unknown
Use similarity search
Known and small
Retrieve the content directly
Known but large
Search within that scope
This does not need to become a complicated routing system. Even a few simple decisions can avoid unnecessary searches and make retrieval easier to understand.
When Similarity Search Is the Right Choice
Similarity search is still one of the most useful parts of RAG. It is especially valuable when the user asks an open question across a large knowledge base, when the application does not know which document contains the answer, or when even a known document contains too much content to send directly to the model.
The important thing is not to use similarity search simply because we are building a RAG system. Use it when there is actually something that needs to be discovered.
Conclusion
Similarity search is excellent at answering one important question.
Which content is most relevant to what the user just asked?
But sometimes our application already knows part of that answer. The user may already be inside a particular document, product, section, or other small area of the knowledge base. If that scope contains only a small amount of information, retrieving it directly can be simpler than running another similarity search.
When the known scope is still large, vector search can remain useful, but we can search inside that smaller space instead of searching everything.
A simple rule is worth remembering.
If you do not know where to look, similarity search can help you find it. If you already know where to look, first ask whether you really need to search again.
