RAG-Based Conversational AI

An end-to-end Retrieval-Augmented Generation (RAG) pipeline that answers questions grounded in a source document, built to reduce hallucination compared to a raw LLM call.

Overview

The system ingests a document (plain text/PDF), splits it into overlapping chunks, embeds those chunks, stores them in a vector database, and retrieves the most relevant chunks at query time to ground an LLM's answer — refusing to answer when the context doesn't contain the information.

Architecture

Document → Text Splitter → Embedding Model → ChromaDB (Vector Store)
                                                     │
User Question ──────────────► Retriever (top-k similarity search)
                                                     │
                                                     ▼
                          Prompt Template (context + question) → LLM → Answer

Tech Stack

• Orchestration: LangChain
• Vector Store: ChromaDB
• Embeddings: OpenAI text-embedding-3-small
• LLM: OpenAI gpt-4.1-nano (swappable with open-source models via HuggingFace)
• Chunking: CharacterTextSplitter — 1,000-character chunks, 200-character overlap

Key Implementation Details

• Grounded answers only: the prompt template explicitly instructs the model to say "I don't know" rather than fabricate an answer when the retrieved context doesn't cover the question.
• Retrieval: top-k=3 similarity search over the vector store.
• Also prototyped: raw ChromaDB collections and cosine-similarity search independently, to understand vector-store internals before building the LangChain-managed pipeline.

Setup

pip install -r requirements.txt
cp .env.example .env   # then add your own OPENAI_API_KEY

Run the notebook RAG_Application.ipynb end to end in Jupyter or Colab.

Note: This project requires your own OpenAI API key. Never commit your .env file or hardcode API keys in source/notebooks — this repo's .gitignore excludes .env by default.

Example

Q: What did the president say about Ketanji Brown Jackson?
A: [Grounded answer generated from the retrieved context of the source document]

Possible Extensions

• Swap OpenAI embeddings/LLM for open-source models (Mistral, Zephyr, Llama) via HuggingFace
• Add PDF/JSON document loaders
• Add conversational memory for multi-turn chat
• Evaluate retrieval quality with RAGAS metrics