Building a RAG based Search System for an Enterprise

There will be only 0.000001% people in tech who have not heard the word RAG. Embed a few documents, find the close ones, paste them into a prompt. The answer comes back. It feels good & easy. You can say it works like this or it works like that. Then I think about the same thing inside a company, and that feeling does not survive.

Same question, two people. They are not allowed to see the same files. If I retrieve first and hide later, the model has already looked. That sat wrong with me. Vector search also misses the boring exact things, error codes, project names, acronyms. And the closest chunk is often not the one that actually answers you. And none of this matters if the model just invents a sentence and puts a citation under it.

So I took a published system design case study. It was written for 5000 users and 500k documents. I did not have that. I had a laptop and free tiers. I still wanted every stage they had. Smaller rooms, same house. I needed to feel the whole path, not a slide of it.

Questions I kept circling:

• How do you make sure a document above someone's clearance never even enters the search?
• Why do we pretend cosine is relevance?
• If I show a live pipeline on the right side of the screen, will I still believe my own architecture?

Ok, let us walk through what a question actually does here.

It first hits some guardrails. Prompt injection, PII, the obvious stuff. If there is chat history, an LLM rewrites the follow-up into a standalone question. Only then I look at cache. Exact cache first, because that one is just a hash of role plus question, no embedding, no drama. Same person, same words, same answer, costs nothing.

If that misses, I embed the question once. One vector, reused everywhere after that. Semantic cache is next, still namespaced by role, so one team's answer cannot leak into another team's mouth. The case study said use 0.95 as the similarity cutoff. With this embedding model, real paraphrases were landing around 0.71. 0.95 almost never fired. I sat with that for a bit. The number on the page was confident. The number in my logs was not. I moved it to 0.90 and stopped pretending their threshold was a law.

Then the search. Two of them, in parallel. Qdrant for meaning. Elasticsearch for the exact ugly strings that vectors forget. Both filters see the user's access levels before they run. Not after. A document above your clearance is not retrieved, not reranked, not shown to Claude. It does not exist for you. That was the part I cared about more than fancy fusion.

The two lists get fused with reciprocal rank fusion, 0.7 on vector, 0.3 on keyword. What does that mean, basically? Both rankings get a vote, closer to the top counts more. Then a cross encoder actually reads the question against the top 50 and keeps 10. This is the slow part. I left it slow on purpose and put the latency on the screen. Hiding it would have made the demo prettier and the system less honest.

Claude writes the answer only from those 10, with citations. Output guardrails check that a citation points at a real chunk. If it does not, it does not ship. Every query is logged.

The UI is a split. Chat on the left. On the right you can watch the path: role, access levels, each stage, how the reranker reordered things, whether cache saved you. I built that right side because I did not want this to be a story I tell. I wanted to look at it while it was happening.

One bug ate a day and I still think about it. The UI was blank. Network tab was full. Events were arriving. I blamed the backend, then the model, then the network. It was my parser. The stream used CRLF, carriage return plus newline, twice between frames. I split on newline newline. They never matched. The whole stream went in. Zero events came out. One line to fix it. The feeling was stupid in a very specific way. Data was right there and my screen was still empty.

For the public URL I use Claude Haiku, rate limit per IP, cap the day, because I do not want a link that quietly burns money. The reranker needs torch and a big local model, so the public build skips it and lives on fusion order. The demo I record has the reranker on. I do not love that split. It is the honest one I could afford.

I keep returning to the same discomfort. A notebook RAG feels finished when the answer sounds right. Inside a company the answer sounding right is the last thing, not the first. Who was allowed to see the file. Did we find the error code or only a cousin of it. Did the model point at a chunk that exists. If those are not true, the sentence does not matter.

That is this artifact for me. Not "I built RAG." I wanted to sit with the parts the demo skips, at a size I could hold, and see which of them still hurt when they were real.