← Projects · Production RAG · Retrieval Engineering

Enterprise Knowledge Q&A: Production RAG

ChromaDB · all-MiniLM-L6-v2 · Azure OpenAI · LangChain · FastAPI

A production RAG system over enterprise policy and procedure documents, with dense semantic retrieval, per-business-unit isolation, cited answers, and an evaluation built on labelled queries.

Problem

Employees spent a long time searching policy manuals and procedures, or interrupting experts for answers that were already written down. Keyword search missed paraphrased questions, and the documents changed frequently, which ruled out fine-tuning. The system needed grounded, cited answers from live documents, and one business unit must never see another unit’s documents.

My role

I built the retrieval layer: embedding model selection, dense retrieval on ChromaDB, per-business-unit isolation through metadata filtering, chunking tuned against a labelled query set, and the retrieval evaluation. Data ingestion and the frontend were owned by other team members.

Tech stack

ChromaDBall-MiniLM-L6-v2Azure OpenAILangChainFastAPI

Key design decisions & trade-offs

  • RAG over fine-tuning. The documents changed frequently; RAG updates in minutes and gives citations, while fine-tuning would lag behind every change.
  • ChromaDB for the first system. An embedded, low-operations vector store fit the corpus size and timeline. At larger scale I would choose a dedicated vector database, which I did in a later system with Qdrant.
  • Isolation by metadata filter. One collection with a mandatory business-unit filter on every query is simpler to operate than a store per unit, while still limiting each caller to their own documents.
  • Self-hosted embeddings. Pinning all-MiniLM-L6-v2 keeps ingestion and query embeddings consistent; a silent model change behind an embeddings API can corrupt an index.
  • Refuse rather than hallucinate. If no retrieved chunk clears the similarity threshold, the system says it does not have enough information instead of calling the LLM.

Evaluation

85%+ Recall@5 on a 200-query labelled set, with P50, P95 and P99 latency monitored across retrieval and end-to-end stages. Recall@5 measures whether the right context was retrieved, not whether the final answer was faithful; answer-level evaluation is what I would add next. The work was recognised with an internal Game Changer of the Month award (September 2023).

Employer and client work is described at a general level. Client identities, internal figures and confidential details are omitted.