LLM System Reliability

Production LLM deployments fail when models answer confidently without sufficient evidence. Most teams discover this in post-mortems, not at design time.

I built this from scratch because specifying RAG systems for regulated clients requires understanding what happens when retrieval returns nothing. The abstention gate is a product decision disguised as an engineering detail.

Complete four-stage RAG pipeline with abstention gate, faithfulness evaluator, and regression harness over a golden query set, zero framework dependencies.

A minimal hand-rolled RAG system in Python: retrieval, confidence estimation, conditional generation, and faithfulness evaluation. No LangChain. Four swappable modules proving the trust layer above the model.

LLMs confidently hallucinate when they lack reliable source material. Most teams bolt on LangChain and hope for the best. This project implements the three reliability primitives of a production LLM pipeline from scratch with zero ML framework dependency: grounded retrieval over authoritative documents, confidence scoring before generation, and explicit abstention when confidence falls below threshold. An engineering study in when an AI system should say I do not know.

1. 1.
   Grounded Retrieval: Keyword search over a curated document corpus. Generation only proceeds from retrieved, authoritative content, not model memory.
2. 2.
   Confidence Scoring + Abstention Gate: Score confidence as min(1.0, len(results) x 0.4). Below 0.5 threshold, refuse to answer rather than guess. The abstention gate is a core production reliability requirement.
3. 3.
   Conditional Generation: When confidence clears threshold, synthesize answer strictly from retrieved documents via a dedicated generation module.
4. 4.
   Faithfulness Evaluation: Measure answer-to-context word overlap as a faithfulness score. Demonstrates awareness of hallucination detection without external eval frameworks.

• ◆
  Modular Pipeline Design: Four independent modules: retrieval.py, abstention.py, generation.py, evaluation.py. Each swappable without breaking the pipeline. Any retrieval backend (vector DB, BM25, hybrid) can slot in later.
• ◆
  No Framework Dependency: Pure Python stdlib plus JSON document store. Built to prove understanding of the trust layer, not to call LangChain wrappers.
• ◆
  Confidence Calibration: Tuned abstention threshold against sample queries where insufficient evidence should produce refusal, not confident wrong answers.
• ◆
  Regression Over Vibes: An eval harness runs a golden query set with acceptance criteria on every change, a drift monitor flags output distribution shifts between model versions, and a failure classifier tags each miss by type and severity.

Consumer AI optimizes for helpfulness. Production AI in regulated contexts must optimize for correctness under uncertainty. The abstention gate is the difference between a system that admits insufficient evidence and one that fabricates an answer with full confidence. This project makes that gate explicit, measurable, and configurable rather than buried inside a framework default.

User query enters retrieval.py for keyword search over sample_docs.json. abstention.py scores confidence from result count. Below 0.5, the system abstains. Above 0.5, generation.py synthesizes from retrieved docs only. evaluation.py computes faithfulness as word overlap between answer and context divided by answer length. Each stage is independently testable and replaceable.