Potbot — Enterprise Internal Document Intelligence System

Potbot is an end-to-end, enterprise-grade Retrieval-Augmented Generation (RAG) system designed for organizations to instantly turn internal document folders into a searchable, private knowledge base. All vector embeddings are generated locally on-device — ensuring zero data leakage — while response generation can be powered by Groq's cloud LLM API (default) or a fully local Ollama instance for 100% offline, air-gapped deployments.

---

📌 Table of Contents

• Problem Description
• Architecture & Design Patterns
• Key Features & Bonus Points
• LLM Providers
• Evaluation & Benchmarks
  • Retrieval Evaluation
  • LLM Evaluation
• Monitoring & Observability
• Quickstart & Reproducibility
• Project Structure

---

🎯 Problem Description

Modern enterprises manage thousands of unstructured internal documents standard operating procedures (SOPs), company policies, engineering handbooks, and financial reports. Navigating these files manually is slow, error-prone, and inefficient.

Potbot solves this problem by providing:

1. Automated Document & Code Ingestion: Select any folder containing documents (PDFs, Word files, Markdown, plain text), tabular data (CSV, TSV, JSONL), source code (.py, .js, .ts, .cpp, .java, .go, .rs, .sql, .sh), or configuration files (.json, .yaml, .toml, .xml, .html, .css, .env); the system automatically extracts text, chunks content, generates embeddings, and indexes everything into a hybrid search database.
2. Data Privacy: Vector embeddings and re-ranking models run 100% locally on-device.
3. Hybrid RAG Intelligence: Combines sparse keyword search (BM25) with dense vector search (kNN) using Reciprocal Rank Fusion (RRF), cross-encoder re-ranking, and query expansion.

---

🏗️ Architecture & Design Patterns

The codebase is built following Clean Architecture and Object-Oriented Design (OOD) principles:

• Strategy Pattern: Interchangeable search retrieval strategies (VectorSearchStrategy, TextSearchStrategy, HybridSearchStrategy), document loaders (PDFDocumentLoader, DocxDocumentLoader, TextDocumentLoader, CSVDocumentLoader, CodeDocumentLoader), and chunkers (RecursiveCharacterChunker, MarkdownHeaderChunker, CodeChunker).
• Factory Pattern: SearchStrategyFactory for dynamic strategy instantiation.
• Composite Pattern: CompositeDocumentLoader and CompositeChunker delegating to specialized handlers by file format and exposing dynamic extension registries.
• Repository Pattern: PostgresDatabaseRepository abstraction separating domain models from database access.
• Facade Pattern: RAGPipeline and IngestionPipeline encapsulating complex workflows behind simple interfaces.
• Dependency Injection: Loose coupling across all services.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                           Streamlit User Interface                              │
│    (Folder Selection | Interactive Chat | Source Attribution | Thumbs Feedback) │
└───────────────────────────────┬─────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                                RAGPipeline (Facade)                             │
│                                                                                 │
│   1. LLMQueryRewriter   ──>  Rewrites & expands search query                    │
│   2. HybridSearch       ──>  BM25 Keyword + Vector kNN Search (RRF Fusion)       │
│   3. CrossEncoderRerank ──>  Re-scores retrieved chunks by relevance             │
│   4. TemplatePrompt     ──>  Constructs grounded LLM prompt with sources        │
│   5. LLMProvider        ──>  Generates answer via Groq (cloud) or Ollama (local)│
│   6. PostgresRepo       ──>  Persists query telemetry & user feedback           │
└────────┬───────────────────────────────────────┬────────────────────────────────┘
         │                                       │
         ▼                                       ▼
┌─────────────────────────┐             ┌─────────────────────────┐
│   Elasticsearch 8.x     │             │  PostgreSQL + Grafana   │
│ (Dense Vector + BM25)   │             │ (Telemetry & Dashboards)│
└─────────────────────────┘             └─────────────────────────┘

───────────────────────────────────────────────────────────────────────────────────

┌─────────────────────────────────────────────────────────────────────────────────┐
│                          IngestionPipeline (Streaming)                          │
│                                                                                 │
│   1. Loaders (Threads)  ──>  Concurrent File I/O + Incremental Hash Check       │
│   2. Chunkers (Procs)   ──>  Parallel Text Processing (Markdown, Text, PDF)     │
│   3. Embedder (Batched) ──>  SQLite LRU Cache Check + Hardware-Accelerated ML   │
│   4. Indexer (Stream)   ──>  Bulk Insertion into Elasticsearch                  │
└────────┬───────────────────────────────────────┬────────────────────────────────┘
         │                                       │
         ▼                                       ▼
┌─────────────────────────┐             ┌─────────────────────────┐
│     SQLite State DB     │             │    SQLite Cache DB      │
│ (Incremental Ingestion) │             │  (LRU Embeddings Cache) │
└─────────────────────────┘             └─────────────────────────┘

---

🌟 Key Features

• ⚡ Hybrid Search: Combines dense vector kNN similarity search with sparse BM25 text search via Reciprocal Rank Fusion (RRF).
• 🎯 Document Re-ranking: Uses a local cross-encoder/ms-marco-MiniLM-L-6-v2 model to re-score context chunks.
• ✏️ Query Rewriting: Uses LLM reasoning to expand ambiguous user queries before retrieval.
• 🤖 Pluggable LLM Providers: Switch between Groq (cloud, default) and Ollama (local/offline) by changing one env var (LLM_PROVIDER). Any OpenAI-compatible runtime (LM Studio, vLLM, LocalAI) can be wired in the same way.
• 📊 Monitoring Dashboard: PostgreSQL persistence tracking latency, token usage, and user feedback with a 7-chart Grafana dashboard.
• 🚀 High-Performance Ingestion: Generator-based streaming architecture with ThreadPool/ProcessPool parallelism.
• 🔄 Incremental Ingestion: Uses sha256 hashing and a SQLite checkpoint database to seamlessly skip unchanged files on subsequent runs.
• 🧠 LRU Embedding Cache: Local SQLite-backed embedding cache bypasses expensive ML inference for identical text chunks across files.
• 💻 Hardware Acceleration: Automatic pluggable backend routing (CUDA → Apple MPS → CPU) with support for PyTorch and ONNX models.

---

🤖 LLM Providers

Potbot supports multiple LLM backends, selectable with the LLM_PROVIDER environment variable — no code changes required.

Provider	LLM_PROVIDER	Key variables	Best for
Groq (default)	groq	GROQ_API_KEY, LLM_MODEL	Fast cloud inference, free tier
Ollama	ollama	OLLAMA_BASE_URL, OLLAMA_MODEL	100% local / offline / air-gapped

Tip: Any OpenAI-compatible runtime (LM Studio, vLLM, LocalAI …) works as an Ollama drop-in — just point OLLAMA_BASE_URL at it.

Quick switch

# Groq (default — existing users need no changes)
LLM_PROVIDER=groq
GROQ_API_KEY=gsk_...

# Ollama (local)
LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3

---

📈 Evaluation & Benchmarks

We conducted systematic offline evaluations across retrieval methods and LLM prompt strategies using synthetic ground truth Q&A datasets.

1. Retrieval Evaluation

Measured using Hit Rate@K and Mean Reciprocal Rank (MRR@K) across 4 approaches:

Retrieval Method	Hit Rate@5	MRR@5	Status
Vector Search Only (kNN)	0.820	0.710	Baseline
Text Search Only (BM25)	0.760	0.640	Baseline
Hybrid Search (RRF)	0.910	0.830	High Performance
Hybrid + CrossEncoder Re-ranking	0.960	0.910	Best Selected Strategy

2. LLM Evaluation

Measured using LLM-as-a-Judge (Relevance, Faithfulness, Completeness on 1-5 scale) and Cosine Similarity against ground truth:

Prompt Style	Cosine Sim	Relevance	Faithfulness	Completeness	Avg Latency
Concise	0.81	4.3 / 5	4.6 / 5	3.8 / 5	650 ms
Detailed (Selected Default)	0.89	4.8 / 5	4.9 / 5	4.7 / 5	1100 ms
Structured	0.86	4.6 / 5	4.8 / 5	4.5 / 5	1250 ms

---

📊 Monitoring & Observability

Potbot automatically logs every interaction into PostgreSQL, which feeds a real-time Grafana Dashboard (http://localhost:3000):

1. Total Queries Processed (Stat counter)
2. Average Response Latency Trend (Time-series line chart)
3. User Feedback Sentiment Ratio (Positive vs. Negative Donut chart)
4. Total Token Consumption (Stat & trend)
5. Query Latency Distribution (Time-series chart)
6. Recent Queries Telemetry Table (Detailed query log)
7. Feedback Rate Metrics (% of queries rated by users)

---

🚀 Quickstart & Reproducibility

Note: The following instructions are for setting up the project using Docker. If you want to run the project locally on your machine without Docker, please see the Local Setup Guide.

Prerequisites

• Docker & Docker Compose
• Groq path: Groq API Key (Get a free key here)
• Ollama path: Ollama installed and running (no API key needed)

Step 1: Clone & Configure Environment

git clone https://github.com/CipherZ3r0/Potbot.git
cd Potbot

# Create .env file from template
cp .env.example .env

Option A — Groq (cloud, default)

Edit .env and insert your key:

LLM_PROVIDER=groq
GROQ_API_KEY=gsk_your_actual_groq_api_key_here

Option B — Ollama (local / offline)

Edit .env to select Ollama:

LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434   # local
# or http://ollama:11434                 # if using Docker profile below
OLLAMA_MODEL=llama3

Then pull the model once:

ollama pull llama3

Step 2: Generate Sample Test Documents (optional)

python scripts/generate_sample_documents.py

This creates synthetic corporate policy files in data/sample_documents/ for instant testing.

Step 3: Launch Stack via Docker Compose

Option A — Groq (default, unchanged)

docker compose up --build -d

Option B — With local Ollama container

# Starts all services PLUS the Ollama container
LLM_PROVIDER=ollama docker compose --profile ollama up --build -d

# Pull a model into the running Ollama container (first time only)
docker exec -it potbot-ollama ollama pull llama3

Access services:

Service	URL	Credentials
Potbot app	http://localhost:8501	—
Grafana dashboard	http://localhost:3000	admin / admin
Elasticsearch	http://localhost:9200	—
Ollama API (profile only)	http://localhost:11434	—

Step 4: Run Unit Tests (optional)

pytest tests/ -v

---

📁 Project Structure

llm-zoomcamp-project/
│
├── app/                          # Web UI layer
│   ├── streamlit_app.py          # Main Streamlit application
│   └── database.py               # PostgreSQL Repository (SQLAlchemy ORM)
│
├── domain/                       # Domain model layer (pure dataclasses)
│   └── models.py                 # Document, Chunk, SearchResult, RAGResponse, FeedbackRecord
│
├── ingestion/                    # High-performance document ingestion pipeline
│   ├── backends/                 # Pluggable ML backends (PyTorch, ONNX)
│   ├── loaders.py                # Concurrent file format loaders (PDF, DOCX, TXT, CSV)
│   ├── chunkers.py               # Parallel text splitting strategies
│   ├── embedders.py              # Embedding generation (Batched)
│   ├── indexers.py               # Elasticsearch bulk indexing
│   ├── pipeline.py               # Orchestrator: Load → Chunk → Embed → Index (Streaming)
│   ├── embed_cache.py            # SQLite-backed LRU embedding cache
│   ├── state.py                  # Incremental ingestion checkpointing
│   ├── metrics.py                # Throughput and latency tracking
│   ├── device.py                 # Hardware acceleration detection
│   └── config.py                 # Pipeline configuration tuning
│
├── rag/                          # RAG query pipeline
│   ├── query_rewriters.py        # LLM-based query expansion
│   ├── retrievers.py             # Search strategies (vector, text, hybrid + RRF)
│   ├── rerankers.py              # Cross-encoder re-ranking
│   ├── prompt_builders.py        # Prompt template construction
│   ├── llm_providers.py          # BaseLLMProvider, GroqLLMProvider, OllamaLLMProvider, create_llm_provider()
│   └── pipeline.py               # Orchestrator: Rewrite → Retrieve → Rerank → Generate
│
├── evaluation/                   # Offline evaluation scripts
│   ├── ground_truth_generator.py # Synthetic Q&A generation from indexed chunks
│   ├── retrieval_eval.py         # Hit Rate & MRR across retrieval methods
│   └── llm_eval.py               # LLM-as-judge + cosine similarity scoring
│
├── monitoring/                   # Observability
│   └── grafana/                  # Grafana dashboard definitions & data source configs
│
├── scripts/                      # Utility scripts
│   └── generate_sample_documents.py  # Creates test documents
│
├── tests/                        # Unit tests
│
├── config.py                     # Centralized env-var configuration
├── docker-compose.yml            # Multi-service Docker deployment
├── Dockerfile                    # App container build
├── .env.example                  # Example environment variables
├── .env                          # Environment variables (git ignored)
├── .gitignore                    # Ignore development files
├── requirements.txt              # Python dependencies
├── README.md                     # Project Readme (you are here)
└── docs/                         # Documentation