RAG Pipeline

A lightweight Retrieval-Augmented Generation (RAG) pipeline that answers user questions from a local knowledge base with grounded answers and citations.

Features

• Document ingestion from .md and .txt files
• Chunking with configurable size and overlap
• FAISS vector index for k-nearest neighbor retrieval
• Google Gemini for answer generation
• Citation validation and grounding checks
• Machine-readable JSON outputs

Architecture

docs/*.md,*.txt  -> Document Loader -> Chunker -> FAISS Index
                                                       |
questions.json  -> Retriever (k=3) -> Gemini Answer Gen -> Citation Validator -> Final Output

Setup

1. Clone and enter the project

cd deriv-rag

2. Create virtual environment

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Set up environment variables

Create a .env file with your Google API key:

GOOGLE_API_KEY=your_api_key_here

Get a free API key from Google AI Studio.

Usage

Run the pipeline

python run.py

This executes all stages:

1. Load documents from docs/
2. Create chunks
3. Build FAISS index with embeddings
4. Load questions from questions.json
5. Retrieve relevant chunks for each question
6. Generate answers using Gemini
7. Validate citations
8. Export final results

Validate artifacts

python validate.py

Checks all generated artifacts for correctness.

Input Files

docs/

Place your knowledge base documents here (.md or .txt files).

questions.json

Questions to answer:

[
  { "id": 1, "question": "What authentication methods does the API support?" },
  { "id": 2, "question": "How long are webhook delivery logs retained?" }
]

Output Artifacts

All outputs are saved to artifacts/:

File	Description
documents.json	Loaded documents with text content
chunks.json	Document chunks with positions
faiss_index.bin	FAISS vector index
chunk_mapping.json	Chunk ID to index mapping
retrieval_results.json	Retrieved chunks per question
answers.json	Generated answers with citations
citation_validation.json	Validation results
final_answers.json	Combined final output
llm_calls.jsonl	LLM call logs

Configuration

Edit constants in run.py:

CHUNK_SIZE = 500       # characters per chunk
CHUNK_OVERLAP = 50     # overlap between chunks
TOP_K = 3              # chunks to retrieve
EMBEDDING_MODEL = "gemini-embedding-001"
GENERATION_MODEL = "gemini-2.5-flash-lite"

Pipeline Stages

INIT
 -> DOCUMENTS_LOADED
 -> CHUNKS_CREATED
 -> INDEX_BUILT
 -> QUESTIONS_LOADED
 -> RETRIEVAL_COMPLETE
 -> ANSWERS_GENERATED
 -> CITATIONS_VALIDATED
 -> RESULTS_EXPORTED
 -> VALIDATION_COMPLETE

Citation Validation

The pipeline validates:

1. Cited chunk IDs exist in retrieved chunks
2. Supported answers have at least one citation
3. Unsupported answers have no citations
4. Numeric claims in answers appear in cited chunks (grounding check)

License

MIT