RAG Q&A System - Django Web Application

A full-featured Django web application built around your exact RAG pipeline from GitHub. Users upload PDFs, the system processes them (PDF → Markdown → Chunks → Embeddings), and anyone can query the documents with natural language questions.

Features

1. PDF Upload & Processing — Dynamic upload with drag & drop. Pipeline runs: PDF → Markdown (marker-pdf) → Chunks (recursive) → Embeddings (all-MiniLM-L6-v2). Duplicate detection via SHA-256 hash.
2. Live Processing Status — Real-time progress tracking: uploaded → extracting → chunking → embedding → ready. Frontend polls every 2 seconds.
3. Shared Document Library — All processed PDFs are stored permanently and shared across all users. Anyone can query any document.
4. Chat With Processed PDFs — Select one or more documents, ask questions. Your exact cosine similarity logic retrieves top chunks and builds the LLM prompt.
5. Persistent Chat History — All sessions and messages saved to SQLite. Come back anytime to review conversations and retrieved chunks.
6. Access Control — Login required to upload and chat. uploaded_by tracked for credit, but documents are queryable by anyone.
7. Reprocess & Delete — Re-run the pipeline or remove bad documents from the library.
8. Multi-Document Chat — Query across 2+ PDFs simultaneously. The retriever searches all selected documents and returns the top-k chunks globally.

Project Structure

django_rag/
├── django_rag/              # Django project settings & URLs
├── rag_core/                # Your exact RAG logic (unchanged)
│   ├── chunk.py             # Recursive chunking (from chunk.py)
│   ├── marker.py            # marker-pdf wrapper (from marker.py)
│   ├── retriever.py         # Cosine similarity retrieval (from RAG.py)
│   ├── pipeline.py          # Orchestrates PDF → Markdown → Chunks → Embeddings
│   └── llm.py               # Configurable LLM backends (TinyLlama, OpenAI, Ollama)
├── accounts/                # User authentication (login, register, logout)
├── documents/               # PDF upload, processing, shared library
├── chat/                    # Chat sessions, messages, querying
├── templates/               # Global templates (base.html, home.html)
├── static/                  # Static files (CSS, JS, uploads)
├── manage.py
├── requirements.txt
└── README.md

Quick Start

1. Install Dependencies

cd django_rag
pip install -r requirements.txt

Note: marker-pdf and sentence-transformers require significant disk space. On first run, the embedding model (all-MiniLM-L6-v2) and TinyLlama will be downloaded automatically.

2. Configure Environment (Optional)

Create a .env file in the project root:

# LLM Backend: 'tinyllama' | 'openai' | 'ollama' | 'custom' | 'prompt_only'
LLM_BACKEND=tinyllama

# For OpenAI:
# OPENAI_API_KEY=sk-...
# LLM_MODEL=gpt-3.5-turbo

# For Ollama:
# LLM_BASE_URL=http://localhost:11434
# LLM_MODEL=llama2

Or edit django_rag/settings.py directly.

3. Run Migrations

python manage.py makemigrations
python manage.py migrate

4. Create a Superuser

python manage.py createsuperuser

5. Run the Server

python manage.py runserver

Visit http://127.0.0.1:8000/ and register a new account, or log in as the superuser.

How It Works

Your Core Logic (Preserved Exactly)

The rag_core package contains your exact algorithms from GitHub, wrapped in importable functions:

• chunk.py → recursive_chunk(text, chunk_size, min_chunk_size) — identical logic from your chunk.py
• marker.py → extract_pdf_to_markdown(pdf_path) — wraps marker_single as you used it
• retriever.py → retrieve_top_chunks(query, basenames, top_k) — your exact cosine similarity logic from RAG.py, extended to support multiple documents
• pipeline.py → process_document(document) — orchestrates the full flow with live status updates

Multi-Document Chat

When you select multiple documents in a chat session, the retriever:

1. Loads chunks and embeddings from all selected documents
2. Combines them into a single array
3. Encodes the query with all-MiniLM-L6-v2
4. Computes cosine distances across all chunks globally
5. Returns the top-k chunks regardless of which document they came from

This is exactly what your RAG.py does, but extended to work with multiple .npy files at once.

Processing Status Polling

Documents update their status in the database as each stage completes:

• pending → uploaded → extracting → chunking → embedding → ready

The frontend polls /documents/<id>/status/ every 2 seconds and shows a live progress tracker with icons for each stage.

LLM Configuration

The chat system is designed to be LLM-agnostic. By default, it returns the built prompt (context + query) so you can see exactly what your RAG pipeline produces. To get actual AI answers, configure one of these backends:

Backend	Setup	Speed	Quality
prompt_only (default)	None	Instant	Shows prompt only
tinyllama	pip install transformers accelerate	Slow (CPU)	Small, local
openai	Set OPENAI_API_KEY	Fast	High quality
ollama	Install Ollama locally	Medium	Depends on model

To change the backend, edit django_rag/settings.py:

LLM_BACKEND = 'openai'  # or 'tinyllama', 'ollama', etc.
LLM_API_KEY = 'sk-...'
LLM_MODEL = 'gpt-3.5-turbo'

Admin Panel

Visit /admin/ to manage:

• Users — see who registered
• Documents — monitor processing status, view hashes, reprocess from admin
• Chat Sessions — review all conversations and retrieved chunks
• Chat Messages — inspect the full prompt + context sent to LLMs

Important Notes

• marker-pdf requires the marker_single command-line tool to be available in your PATH after installation.
• First run will download the all-MiniLM-L6-v2 embedding model (~80MB) to ~/.cache/torch/sentence_transformers/.
• TinyLlama requires ~2GB download on first use if configured.
• File uploads are stored in media/uploads/pdfs/ and processed files in static/uploads/.
• The project uses SQLite by default for simplicity. For production, switch to PostgreSQL.

Differences from Your Original Repo

Your GitHub Repo	This Django App
Hardcoded file paths	Dynamic file paths from DB
Single-query script	Web UI with chat sessions
No persistence	SQLite DB for everything
No auth	Login required
One document at a time	Multi-document chat
No status visibility	Live polling progress
Local only	Web-accessible

Your core algorithms (chunking, embedding, cosine retrieval, prompt building) are 100% preserved in rag_core/. "# RagWebApplication"