← Projects · Production RAG · Retrieval Engineering

Enterprise Knowledge Q&A — Production RAG

ChromaDB · all-MiniLM-L6-v2 · Azure OpenAI · LangChain · FastAPI

A production RAG system over enterprise document libraries across banking, retail, and financial-services business units — dense semantic retrieval with metadata-filtered per-unit isolation, token-budgeted prompts with mandatory citations, and an evaluation framework built on labelled queries rather than vibes.

Problem

Employees averaged 15 minutes per query hunting through policy manuals, compliance guidelines, and SOPs — or interrupting subject-matter experts for answers that were already documented. Keyword search missed paraphrased questions; the knowledge base changed weekly, ruling out fine-tuning. The system needed grounded, cited answers from live documents, with strict per-business-unit isolation so one unit could never surface another’s documents.

My role

I built the RAG and retrieval components of the system: the ChromaDB dense-retrieval design, embedding-model selection, the metadata-filtered per-business-unit isolation, chunking and token-budget tuning, and the labelled-query evaluation framework. It reached production, serving 200+ users across three business units. Data-engineering ingestion and the frontend were owned by other team members.

Architecture

Ingestion (offline)docs → semantic chunking (sentence-aware, tuned overlap)→ all-MiniLM-L6-v2 embeddings → ChromaDB (per-BU metadata) User query+ business-unit id ChromaDB dense top-kcosine over MiniLM vectorsmetadata filter = caller’s BU Token-budgeted prompt → Azure OpenAIlost-in-the-middle ordering · few-shot citationslow temperature · mandatory source citations Confidence gatebelow threshold → “I don’thave enough information” Serving & monitoringFastAPI · Redis sessions · DockerizedP50/P95/P99 latency, retrieval + end-to-end Dense, single-store retrieval: one ChromaDB collection, per-business-unit isolation enforced by a metadata filter on every query — a caller can only ever retrieve their own unit’s documents.

Dense retrieval, tuned for the domain. Queries and chunks share one embedding space (all-MiniLM-L6-v2); retrieval is cosine similarity over ChromaDB, with a metadata filter pinning every query to the caller’s business unit. Semantic, sentence-aware chunking keeps policy clauses and procedure steps intact, and the chunk-size/overlap was tuned against a labelled query set rather than guessed.

Grounded generation, measured. Retrieved chunks feed a token-budgeted prompt — highest-relevance chunks placed first and last to counter lost-in-the-middle, with few-shot citation examples so every answer carries its sources. Prompt and chunking tuning together cut token cost ~20%. A confidence gate refuses to answer when no chunk clears the similarity threshold, rather than letting the model improvise.

Tech stack

ChromaDBall-MiniLM-L6-v2Azure OpenAILangChainHuggingFace / sentence-transformersFastAPIRedisDockerPyTorch

Key design decisions & trade-offs

  • RAG over fine-tuning. Knowledge changed weekly; retraining costs thousands per run and takes hours, while RAG updates in minutes and produces citations for free.
  • ChromaDB over Qdrant/Pinecone. An embedded, zero-ops vector store fit the corpus size and timeline — an acknowledged trade-off I’d make differently at 5× scale (and did, moving to Qdrant in a later system).
  • Metadata filtering for isolation. One collection with a per-business-unit metadata filter on every query — simpler to operate than a store per unit, while still guaranteeing a caller only ever sees their own documents.
  • Self-hosted embeddings over an embeddings API. Pinning all-MiniLM-L6-v2 guarantees ingestion/query consistency — an API-side model change can silently poison a vector index — and keeps embedding free and local.
  • Refuse over hallucinate. If every retrieved chunk falls below the similarity threshold, the system says “I don’t have enough information” instead of calling the LLM at all.

Results

85%+ Recall@5 (200-query labelled set) · ~20% token-cost reduction · 200+ users across 3 business units · retrieval cut from ~15 min to ~30 sec

P50/P95/P99 latency monitoring ran across the retrieval and end-to-end stages. The system reached production and was recognized with an internal Game Changer of the Month award (September 2023).