Multimodal RAG System 2025

A production-grade, multimodal Retrieval-Augmented Generation (RAG) platform that enables users to upload heterogeneous document types (PDFs, Word documents, images, and audio files), store them locally in a high-performance vector store, and perform grounded natural language querying with exact source attribution down to physical page numbers and audio timestamps.

---

📋 Table of Contents

• Overview
• Key Features
• How It Works
• System Architecture
• Supported Document Formats
• Tech Stack
• Project Directory Structure
• Prerequisites
• Environment Configuration
• Installation & Local Setup
• Running the Application
• API Documentation
• Grounding & Safety Rules
• Troubleshooting & Gotchas
• Development & Evaluation
• Future Roadmap
• License

---

🎯 Overview

Traditional RAG implementations are often limited to raw plain text processing. Multimodal RAG System 2025 bridges the gap across formats by extracting structural text from scanned PDFs, formatted DOCX spreadsheets, static images, and spoken voice recordings.

It combines Qdrant Vector Search, Lexical Term Search, Reciprocal Rank Fusion (RRF), Cross-Encoder Reranking, and Google Gemini 3.6 Flash into an integrated, zero-hallucination research pipeline.

---

✨ Key Features

• 📂 Native Multimodal Ingestion: Ingest text, layout-heavy PDFs, Microsoft Word tables, scanned image text, and multi-format audio streams.
• ⚡ Dual-Engine Retrieval: Concurrent semantic embedding search and full-text keyword matching powered by Qdrant.
• 🔀 Reciprocal Rank Fusion (RRF): Merges vector search and term matching without needing score normalization.
• 🎯 Cross-Encoder Reranking: Applies ms-marco-MiniLM-L-6-v2 cross-encoder scoring to filter candidate chunks down to the highest relevance set.
• 📌 Exact Provenance & Smart Citations: Citation manager maps answers back to exact physical PDF pages and audio timestamps. Unused candidate chunks are filtered out automatically.
• 🛡️ Strict Grounding & Refusal Logic: System prompt guidelines restrict answers to retrieved context. Unknown topics trigger standard refusal responses.
• 🔄 Auto-Deduplication: Uploading updated document versions automatically purges stale vector embeddings by filename.

---

⚙️ How It Works

1. Document Upload & Ingestion: Files uploaded via the API or Web UI are routed to specialized extractors (PDF parser, DOCX extractor, Tesseract OCR, or Whisper Transcriber).
2. Chunking & Embedding: Extracted text is broken into 800-word sliding windows with overlapping boundaries and converted into 384-dimensional dense vectors using paraphrase-multilingual-MiniLM-L12-v2.
3. Storage: Chunks and payloads (metadata like filename, page number, timestamp) are stored in an embedded Qdrant instance.
4. Query & Retrieval: User questions are analyzed for modality hints and executed concurrently across vector index and keyword filters.
5. Reranking & Context Assembly: RRF merges top candidates, Cross-Encoder reranks them, and the Context Builder constructs a clean prompt context.
6. Answer Synthesis: Gemini 3.6 Flash generates a grounded answer with inline numerical markers [1], [2].

---

🏗️ System Architecture

flowchart TD
    subgraph Client
        UI[Vanilla JS / HTML5 Frontend]
    end

    subgraph API Layer
        FA[FastAPI App Assembly]
        DS[Document Service]
        QS[Query Service]
    end

    subgraph Ingestion Engine
        IP[Ingestion Pipeline]
        PDF[PDF Loader - pypdfium2]
        DOCX[DOCX Loader - python-docx]
        OCR[Image OCR - Tesseract]
        STT[Speech-to-Text - Whisper]
        CHK[Text Chunker - 800 Words]
        EMB[Embedding Provider - MiniLM-L12]
    end

    subgraph Storage & Retrieval
        QD[(Qdrant Vector Database)]
        HYB[Hybrid Retriever - Vector + Keyword]
        RRF[Reciprocal Rank Fusion]
        CE[Cross-Encoder Reranker]
    end

    subgraph Generation & Attribution
        PM[Prompt Manager]
        LLM[Google Gemini 3.6 Flash]
        CM[Citation Manager]
    end

    UI -->|POST /api/documents/upload| FA
    UI -->|POST /api/query| FA

    FA --> DS & QS
    DS --> IP
    IP --> PDF & DOCX & OCR & STT
    PDF & DOCX & OCR & STT --> CHK --> EMB --> QD

    QS --> HYB
    HYB -->|Vector Search| QD
    HYB -->|Keyword Search| QD
    HYB --> RRF --> CE --> PM
    PM --> LLM --> CM --> UI

Loading

---

📄 Supported Document Formats

Category	File Extensions	Processing Pipeline / Model	Output Metadata
PDF Documents	.pdf	pypdfium2 with paragraph re-assembly & OCR fallback	Page number, Paragraph index
Word Documents	.docx	python-docx for headings, text, and tables	Paragraph index, Section ID
Image Files	.png, .jpg, .jpeg	Tesseract OCR engine	Modality tag, Document ID
Audio Tracks	.mp3, .wav, .m4a, .aac	OpenAI Whisper (base model)	Timestamp start/end (MM:SS)

---

🛠️ Tech Stack

Backend Infrastructure

• Framework: FastAPI (Python 3.11+)
• Vector Database: Qdrant (Embedded local storage or remote cluster)
• Embedding Model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (384-dim)
• Reranking Engine: cross-encoder/ms-marco-MiniLM-L-6-v2
• Generative AI Provider: Google Gemini API (google-genai SDK, Model: gemini-3.6-flash)
• Speech Recognition: OpenAI Whisper
• OCR Engine: Tesseract OCR
• PDF & Document Parsers: pypdfium2, python-docx

Frontend Application

• Stack: Vanilla HTML5, Modern CSS Design System (Glassmorphism & Micro-animations), Vanilla JavaScript (ES6 Modules)
• Communications: Async Fetch API

---

📁 Project Directory Structure

multimodal-rag/
├── app/
│   ├── api/                    # FastAPI routes, Pydantic schemas, dependency container
│   │   ├── routes/             # Endpoint definitions (documents, query, chat, health)
│   │   ├── schemas/            # Request and Response Pydantic models
│   │   └── dependencies.py     # Singleton container initialization
│   ├── chunking/               # Sliding-window text splitter
│   ├── core/                   # Global settings, logging configuration, custom exceptions
│   ├── embeddings/              # Embedding model wrappers
│   ├── extraction/              # Tesseract OCR & Whisper Speech-to-Text wrappers
│   ├── ingestion/               # Loaders for PDF, DOCX, Images, and Audio
│   ├── llm/                     # Gemini provider abstraction
│   ├── rag/                     # RAG orchestrator, prompt builder, citation manager
│   ├── retrieval/               # Hybrid retriever, RRF fusion, Cross-Encoder reranker
│   ├── services/                # Business logic (DocumentService, QueryService)
│   ├── utils/                   # Helper functions (file management, timing)
│   ├── vectorstore/             # Qdrant VectorStore interface and implementation
│   └── main.py                  # Server entrypoint and middleware definition
├── data/                        # Persistent local data directory
│   ├── uploads/                 # Temporary upload staging
│   ├── processed/               # Extracted assets
│   └── qdrant/                  # Local Qdrant database storage
├── frontend/                    # Embedded single-page web UI
│   ├── css/                     # Main CSS styling
│   ├── js/                      # App logic and event handlers
│   └── index.html               # Main user interface
├── tests/                       # Automated test suite
│   ├── unit/                    # Unit tests for chunking, citations, and metadata
│   └── integration/             # Integration tests for ingestion and pipeline
├── Dockerfile                   # Production container definition
├── docker-compose.yml           # Container orchestration file
├── pytest.ini                   # PyTest configuration
└── requirements.txt             # Python dependencies

---

⚡ Prerequisites

Before installing the project, make sure the following dependencies are installed on your environment:

1. Python 3.11+
2. Tesseract OCR:
   • Windows: Download installer from UB-Mannheim GitHub and add to System PATH.
   • Linux: sudo apt-get install tesseract-ocr
   • macOS: brew install tesseract
3. FFmpeg: (Required by Whisper for audio processing)
   • Linux: sudo apt-get install ffmpeg
   • macOS: brew install ffmpeg
   • Windows: Install via winget install ffmpeg or download binaries.
4. Google Gemini API Key: Obtain a free API key at Google AI Studio.

---

🔑 Environment Configuration

Create a .env file in the project root directory by copying .env.example:

cp .env.example .env

Set your configuration values inside .env:

# --- App Settings ---
APP_NAME=Multimodal RAG System
ENV=development
PORT=8000

# --- Vector Store ---
QDRANT_PATH=data/qdrant
QDRANT_COLLECTION=multimodal_rag

# --- AI Models ---
EMBEDDING_MODEL=paraphrase-multilingual-MiniLM-L12-v2
RERANKER_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
GEMINI_MODEL=gemini-3.6-flash

# --- Credentials ---
GEMINI_API_KEY=your_gemini_api_key_here

---

🚀 Installation & Local Setup

1. Clone Repository

git clone https://github.com/ONKAR-100/Multimodal_RAG_System_.git
cd multimodal-rag

2. Set Up Virtual Environment

On Windows (PowerShell):

python -m venv venv
.\venv\Scripts\Activate.ps1

On macOS / Linux:

python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

pip install --upgrade pip
pip install -r requirements.txt

---

🏃 Running the Application

Local Development Server

Run FastAPI using Uvicorn:

uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

Access points:

• Web UI: http://127.0.0.1:8000
• Interactive Swagger API Docs: http://127.0.0.1:8000/docs
• ReDoc API Documentation: http://127.0.0.1:8000/redoc

Docker Deployment

To launch using Docker Compose:

export GEMINI_API_KEY="your_api_key_here"
docker-compose up --build

---

🌐 API Documentation

1. Document Upload

POST /api/documents/upload Uploads and indexes a single document file.

curl -X POST "http://127.0.0.1:8000/api/documents/upload" \
  -H "accept: application/json" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@sample_syllabus.pdf"

2. Query Documents

POST /api/query Submits a question to the RAG pipeline.

curl -X POST "http://127.0.0.1:8000/api/query" \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the course code for Machine Learning?"}'

3. List Documents

GET /api/documents Lists all indexed documents.

4. Delete Document

DELETE /api/documents/{document_id} Deletes a specific document by ID.

5. Clear Knowledge Base

DELETE /api/documents Purges all indexed vectors and documents from the collection.

---

🛡️ Grounding & Safety Rules

To prevent hallucinations, the system relies on strict rules in app/rag/prompt_manager.py:

1. Answers must be derived only from retrieved document context.
2. Every claim must include bracketed numerical citations like [1].
3. If retrieved evidence does not contain sufficient details, the system returns:
   "I cannot answer this question based on the provided documents."

---

❓ Troubleshooting & Gotchas

• Initial Startup Delays: On the first launch, downloading sentence-transformers, CrossEncoder, and Whisper weights can take 1–2 minutes.
• Browser UI Caching: If JavaScript changes are not showing in the browser, perform a Hard Refresh (Ctrl + F5 or Cmd + Shift + R).
• FFmpeg Error on Audio Ingestion: Ensure ffmpeg is accessible from your system command line (ffmpeg -version).
• Physical vs Printed Page Numbers: Citations report physical PDF page indices (1-indexed) so you can directly navigate pages in PDF viewers.

---

🧪 Development & Evaluation

To run unit and integration tests:

pytest tests/unit -v
pytest tests/integration -v

---