← Projects · RAG · Retrieval Engineering

Enterprise Knowledge Q&A — RAG System

ChromaDB · Elasticsearch · Cross-encoder reranking · Azure OpenAI · LangChain

A RAG system over enterprise document libraries across banking, retail, and financial-services business units — hybrid dense+sparse retrieval with cross-encoder reranking, token-budgeted prompts with mandatory citations, and an evaluation framework built on labelled queries rather than vibes.

Problem

Employees averaged 15 minutes per query hunting through policy manuals, compliance guidelines, and SOPs — or interrupting subject-matter experts for answers that were already documented. Keyword search missed paraphrased questions; the knowledge base changed weekly, ruling out fine-tuning. The system needed grounded, cited answers from live documents, at enterprise scale, with per-business-unit isolation.

My role

I worked on the ML/GenAI side as part of the project team: the hybrid retrieval design (ChromaDB + Elasticsearch), cross-encoder reranking, prompt and token-budget engineering, and the labelled-query evaluation framework. Document-ingestion data engineering, the frontend, and overall project leadership sat with other team members.

Architecture

Ingestion (offline, nightly rebuild)docs → semantic chunking (300–500 tokens, sentence-aware)→ HuggingFace embeddings → ChromaDB + Elasticsearch User queryrewrite if ambiguous ChromaDB · dense top-k Elasticsearch · BM25 RRF fusionrank-based merge Cross-encoder reranktop 20–50 → top 5 Token-budgeted prompt → Azure OpenAIlost-in-the-middle placement · few-shot citationstemp 0.1–0.2 · mandatory source citations Guardrails & servingconfidence gate → “I don’t know” below thresholdper-BU vector-store isolation (JWT) · Redis sessions · FastAPI Monitoring: P50/P95/P99 latency per stage (retrieval / rerank / generation) · approval rate · weekly hallucination spot-checks · token spend with anomaly alerts.

Why hybrid retrieval. The dense-only first version plateaued at 65–70% recall — semantic similarity kept missing exact identifiers like policy codes and regulation citations. Adding BM25 sparse retrieval and merging via Reciprocal Rank Fusion (rank-based, so the two systems never need a shared score scale) is what pushed recall past 85%. Neither system alone would have gotten there.

Two-stage reranking. A bi-encoder narrows millions of chunks to ~50 candidates cheaply; a cross-encoder — 10–50× slower but far more accurate — re-orders only that shortlist. The top 5 enter a token-budgeted prompt with highest-relevance chunks placed first and last to mitigate lost-in-the-middle.

Tech stack

ChromaDBElasticsearchCross-encoder (ms-marco)Azure OpenAILangChainHuggingFace / sentence-transformersFastAPIRedisDockerPyTorch

Key design decisions & trade-offs

  • RAG over fine-tuning. Knowledge changed weekly; retraining costs thousands per run and takes hours, while RAG updates in minutes and produces citations for free.
  • ChromaDB over Qdrant/Pinecone. An embedded, zero-ops vector store fit the corpus size and timeline — an acknowledged trade-off I’d make differently at 5× scale (and did, moving to Qdrant in a later system).
  • Self-hosted embeddings over an embeddings API. Pinning the exact model version guarantees ingestion/query consistency — an API-side model change can silently poison a vector index.
  • Semantic chunking over fixed-size. Policy clauses and procedure steps survive intact; 300–500-token chunks with sentence-boundary grouping measured best on recall.
  • Refuse over hallucinate. If every retrieved chunk falls below the similarity threshold, the system says “I don’t have enough information” instead of calling the LLM at all.

Results & honest scope

85%+ recall@5 — measured on 200 manually-labelled queries across 3 business units · dense-only baseline plateaued at 65–70% before hybrid retrieval closed the gap

The system reached internal pilot use with real users while I was there: answers that previously took ~15 minutes of manual document hunting came back in well under a minute. I left the company before a full production rollout, so I don’t claim post-exit outcomes — the retrieval-quality numbers above are what I measured myself.