Explainable Corrective RAG Platform

A production-style Corrective Retrieval-Augmented Generation project with retrieval quality evaluation, adaptive routing, sentence-level evidence refinement, source citations, a Streamlit demo, a FastAPI endpoint, and RAGAS benchmarking.

The project demonstrates how to move beyond basic "retrieve and answer" RAG by adding a corrective layer that decides whether retrieved context is strong enough to trust, whether to fall back to web search, or whether to combine internal and external evidence.

Highlights

• Corrective RAG pipeline inspired by the CRAG paper.
• FAISS vector search over PDF documents.
• OpenAI embeddings and chat models.
• LLM-based chunk relevance evaluation.
• Lexical relevance prior blended with LLM scoring.
• MMR retrieval to reduce duplicate context.
• Dynamic routing across CORRECT, INCORRECT, and AMBIGUOUS paths.
• Sentence-level context filtering before answer generation.
• Source metadata and citation-ready outputs.
• Streamlit interface for interactive demos.
• FastAPI endpoint for application integration.
• RAGAS evaluation across faithfulness, answer relevancy, context precision, and context recall.

Results

Evaluation was run on a 10-question golden dataset built from the "Attention Is All You Need" paper.

Variant	Faithfulness	Answer Relevancy	Context Precision	Context Recall
Basic RAG	0.9500	0.9919	0.7667	0.9000
CRAG Correct	0.8333	0.8879	0.9000	0.9000
CRAG Full	0.9000	0.9901	0.9000	0.8500
Optimized CRAG	0.9467	0.9836	0.9000	0.9000

The optimized project pipeline keeps the higher context precision of the corrective variants while recovering basic-RAG-level recall and near-basic faithfulness.

Architecture

PDF documents
  -> chunking
  -> OpenAI embeddings
  -> FAISS vector index
  -> MMR candidate retrieval
  -> LLM + lexical relevance scoring
  -> route:
       CORRECT   -> internal evidence
       INCORRECT -> web fallback
       AMBIGUOUS -> internal + web evidence
  -> sentence-level evidence filtering
  -> grounded answer generation
  -> answer + sources + trace

The projectized implementation lives in src/crag.

Repository Structure

.
|-- src/crag/                      # Reusable CRAG package
|   |-- ingestion.py               # PDF loading and chunking
|   |-- retrieval.py               # FAISS, MMR, retrieval helpers
|   |-- pipeline.py                # Main CorrectiveRAGPipeline
|   |-- prompts.py                 # Prompt templates
|   |-- schemas.py                 # Typed outputs and result objects
|   |-- settings.py                # Runtime configuration
|   `-- text_scoring.py            # Lightweight lexical scoring
|-- app.py                         # Streamlit demo
|-- api.py                         # FastAPI service
|-- evaluate_project.py            # Optimized pipeline RAGAS evaluation
|-- evaluate_crag.py               # Original notebook-style variant evaluation
|-- eval_data/                     # Golden dataset and evaluation PDF
|-- documents/                     # Source PDFs for notebook demos
|-- tests/                         # Lightweight unit tests
|-- 1_basic_rag.ipynb              # Basic RAG notebook
|-- 2_retrieval_refinement.ipynb   # Sentence-level refinement
|-- 3_retrieval_evaluator.ipynb    # Retrieval scoring and routing
|-- 4_web_search_refinement.ipynb  # Web fallback
|-- 5_query_rewrite.ipynb          # Query rewrite before web search
`-- 6_ambiguous.ipynb              # Internal + web handling for ambiguous retrieval

Pipeline Variants

Variant	Behavior
basic	Retrieves top-4 chunks and answers directly.
correct	Scores chunks, uses internal docs for CORRECT, web fallback for INCORRECT, and returns an ambiguity message for AMBIGUOUS.
full	Handles AMBIGUOUS by combining internal and web evidence.
optimized	Uses MMR retrieval, broader candidate search, LLM + lexical scoring, adaptive evidence selection, citations, and stricter answer generation.

Setup

Use Python 3.10+. Modern LangChain and LangGraph do not install reliably on Python 3.8.

Option 1: venv

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .

On Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .

Option 2: Conda

conda create -n crag310 python=3.10 -y
conda activate crag310
pip install -r requirements.txt
pip install -e .

Environment Variables

Create a .env file from the example:

cp .env.example .env

Then add:

OPENAI_API_KEY=your_openai_api_key
TAVILY_API_KEY=your_tavily_api_key_optional

OPENAI_API_KEY is required for embeddings, generation, and RAGAS evaluation. TAVILY_API_KEY is only needed for web-search fallback paths.

Never commit .env or API keys to GitHub.

Run The Streamlit Demo

streamlit run app.py

The demo shows:

• final answer
• retrieval verdict
• route taken by the pipeline
• selected sources
• candidate chunk scores
• refined evidence context

Run The API

uvicorn api:app --reload

Health check:

curl http://127.0.0.1:8000/health

Ask a question:

curl -X POST http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question":"How many attention heads does the base Transformer use?"}'

Windows PowerShell:

curl.exe -X POST http://127.0.0.1:8000/ask `
  -H "Content-Type: application/json" `
  -d "{\"question\":\"How many attention heads does the base Transformer use?\"}"

Run Evaluation

Run the optimized project evaluation:

python evaluate_project.py

This writes:

eval_results_crag_optimized.json

Run the original notebook-style comparison:

python evaluate_crag.py

This writes:

eval_results_crag_basic.json
eval_results_crag_correct.json
eval_results_crag_full.json

Run Tests

python -m pytest -q -p no:cacheprovider

The lightweight tests cover dependency-free scoring utilities. Full pipeline behavior requires OpenAI API access.

Notebooks

The notebooks show the step-by-step progression from basic RAG to corrective RAG:

1. 1_basic_rag.ipynb: basic PDF retrieval and generation.
2. 2_retrieval_refinement.ipynb: sentence-level filtering.
3. 3_retrieval_evaluator.ipynb: chunk scoring and routing.
4. 4_web_search_refinement.ipynb: web fallback.
5. 5_query_rewrite.ipynb: query rewriting for web search.
6. 6_ambiguous.ipynb: hybrid internal + web evidence for ambiguous retrieval.