Offline RAG Agent: Document QA over Expert Call Transcripts

A retrieval-augmented generation pipeline that answers questions grounded in specific documents — built entirely with local models, no API keys or cloud dependencies.

What It Does

Takes 10 expert call transcripts (PDFs from Uber), chunks and embeds them into a local vector store, and uses a quantized Mistral 7B model to answer natural language questions using only the retrieved context. Runs completely offline.

Why I Built It This Way

I chose to run everything locally — local embeddings, local vector DB, local LLM — because I wanted to understand the complete pipeline rather than wrapping an OpenAI API call. Every component is something I can inspect, debug, and swap out.

Architecture

PDF Transcripts
    → PyMuPDF text extraction
    → JSON intermediate format
    → LangChain recursive chunking (500 chars, 100 overlap)
    → MiniLM embedding (SentenceTransformers)
    → ChromaDB vector storage

User Query
    → MiniLM embedding
    → ChromaDB similarity search (top-5)
    → Retrieved chunks + system prompt
    → Mistral 7B (4-bit GGUF, llama-cpp-python)
    → Answer

The Hallucination Problem

The model initially conflated details between experts who had similar roles at Uber — attributing one person's statements to another. Root cause: semantically similar chunks from different transcripts were being retrieved together, and the model filled in gaps.

What I did about it:

• Added metadata filtering so retrieval can be scoped to a single transcript
• Reduced temperature to minimize creative completions
• Strengthened system prompt to constrain answers to retrieved context only
• Added explicit fallback: "The answer is not available in the provided transcript."
• Documented all observed hallucination patterns in hallucination_report.md

Stack

Component	Tool	Why
LLM	CapybaraHermes / Mistral 7B (Q4_K_M GGUF)	Small enough to run locally, good instruction-following
Embeddings	all-MiniLM-L6-v2	Fast, accurate for semantic similarity
Vector DB	ChromaDB	Persistent local storage, metadata filtering support
Chunking	LangChain RecursiveCharacterTextSplitter	Handles document structure well
Runtime	llama-cpp-python	Efficient local inference for GGUF models
UI	Streamlit	Simple web interface for querying

How to Run

# Setup
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Download the model (~4.4GB)
# Place in models/local_model/capybarahermes-2.5-mistral-7b.Q4_K_M.gguf

# Build the vector store (one-time)
python vector_store.py

# Run the app
streamlit run run.py

Project Structure

├── run.py                  # Streamlit entry point
├── app/
│   └── rag_pipeline.py     # Retrieval + generation logic
├── vector_store.py         # Chunking, embedding, ChromaDB ingestion
├── test_llm.py             # Standalone LLM chat for testing
├── data/
│   └── transcripts_json/   # Extracted transcript text (JSON)
├── models/
│   └── local_model/        # GGUF model files (not tracked in git)
├── vector_db/              # ChromaDB persistent storage
├── hallucination_report.md # Documented failure modes
├── requirements.txt
└── README.md

What I'd Improve

• Add a cross-encoder re-ranker after initial retrieval
• Add BM25 hybrid search alongside dense embeddings
• Batch the embedding step (currently processes one chunk at a time)
• Build a proper evaluation set with known answers to measure accuracy quantitatively
• Add source attribution in the UI so users can verify which chunk the answer came from

Key Takeaway

Retrieval quality matters more than model quality. Better retrieval consistently improved answer accuracy more than swapping to a different LLM. The model can only be as good as the context it receives.