LocalRAG

A local-first RAG knowledge base for personal documents. Upload PDF/Markdown files and ask questions in natural language — every answer is streamed (SSE) and grounded in citations that link back to the exact source page and excerpt.

Works entirely on your machine with local models (Ollama / vLLM), or plug in any OpenAI-compatible cloud API (DeepSeek, SiliconFlow, etc.). Also exposes an OpenAI-compatible API and MCP server so AI agents (opencode, cherrystudio, codex, …) can query your knowledge base.

Features

• Document management — upload PDF / Markdown, automatic chunking & vectorization, delete documents, track ingestion status
• Citation-grounded Q&A — retrieval-augmented streaming answers (SSE) with multi-turn conversation and adjustable Top-K
• Source tracing — every answer comes with source cards (document, page, verbatim excerpt) that jump to the cited location
• Local-first & private — runs fully offline with Ollama / vLLM; no data leaves your machine unless you opt into a cloud API
• Provider-agnostic — LLM and Embedding both go through OpenAI-compatible interfaces; switch between local and cloud via config
• Agent-ready — OpenAI-compatible API (/v1/chat/completions) + MCP server (retrieve_documents, rag_answer)
• Single-port deployment — the FastAPI backend serves the Vue 3 frontend build (web/dist); no extra static server needed
• Settings panel — inspect Qdrant health and the currently active model configuration

Architecture

Browser ── HTTP/SSE ──> FastAPI (app/) ──> Qdrant (vector search)
                          │
                          ├──> SQLite (conversations / document metadata)
                          └──> LLM / Embedding provider (OpenAI-compatible, configurable)

• Backend: FastAPI + SQLAlchemy (SQLite) + Qdrant vector store
• Frontend: Vue 3 + Element Plus + Vite (build artifacts served by the backend)
• Providers: LLM & Embedding via OpenAI-compatible endpoints — local (Ollama / vLLM) or cloud (DeepSeek, SiliconFlow, …)

Quick Start

Prerequisites: Python 3.11+, Node.js 18+, Docker (or an existing Qdrant service).

1. Start the vector store

Qdrant via Docker:

docker compose up -d

Or run Qdrant natively (download the qdrant binary), default port localhost:6333.

2. Install dependencies

Backend:

conda create -n llm python=3.11 -y
conda activate llm
pip install -e ".[dev]"

Frontend build:

cd web
npm install
npm run build
cd ..

web/dist must exist for the backend to serve the UI; during development run npm run dev in another terminal for hot reload.

3. Configure models

Copy-Item .env.example .env

Edit .env:

• KB_CONFIG=config/local.yaml — local Ollama (pull the models first: ollama pull qwen2.5:7b-instruct, ollama pull bge-m3)
• KB_CONFIG=config/cloud.yaml — cloud APIs (fill in KB_LLM_API_KEY, KB_EMBEDDING_API_KEY)

4. Run

python -m uvicorn app.main:app --host 0.0.0.0 --port 8000

Open http://localhost:8000.

Configuration

Configuration has two layers: config/*.yaml presets (local.yaml for Ollama, cloud.yaml for cloud APIs), and KB_* environment variables in .env which override the presets:

Variable	Meaning
KB_CONFIG	Config preset: config/local.yaml or config/cloud.yaml
KB_LLM_BASE_URL / KB_LLM_API_KEY / KB_LLM_MODEL	LLM endpoint, key, model
KB_EMBEDDING_BASE_URL / KB_EMBEDDING_API_KEY / KB_EMBEDDING_MODEL	Embedding endpoint, key, model
KB_QDRANT_URL	Qdrant address (default http://localhost:6333)

Security: put API keys only in .env (git-ignored) — never commit them to config/*.yaml or the repository.

Using Local Models

Every model is accessed through an OpenAI-compatible endpoint, so local weights only need to be exposed as one:

• vLLM: vllm serve Qwen3-8B --served-model-name Qwen3-8B (default http://localhost:8000/v1), then in .env:

  KB_CONFIG=config/local.yaml
  KB_LLM_BASE_URL=http://localhost:8000/v1
  KB_LLM_MODEL=Qwen3-8B
  

• Ollama: ollama run qwen2.5:7b-instruct (default http://localhost:11434/v1); for embeddings run ollama pull bge-m3 and set KB_EMBEDDING_MODEL=bge-m3.

Embedding works the same way — point it at the OpenAI-compatible endpoint of your local embedding service.

Agent Integration

The service exposes an OpenAI-compatible API and an MCP server for tools like opencode / codex / cherrystudio (loopback only, no auth).

Note: /v1/* has no authentication. Bound to 127.0.0.1 for local-only use; do not expose to the network without adding auth.

OpenAI-compatible API

• Base URL: http://127.0.0.1:8000/v1
• Models:
  • rag-chat — retrieves from the knowledge base and answers with citations
  • qwen3-8b — raw local model passthrough (no retrieval)

MCP

• Command: python -m app.mcp_server (run in the repo root)
• Tools: retrieve_documents(query, top_k), rag_answer(question)

Tests

python -m pytest -v

Covers chunking, ingestion, vector retrieval, citation generation, SSE streaming, and the full API chain (integration tests use a mock provider and Qdrant :memory:, no real models or services required).

Troubleshooting

• Page won't open after start? Make sure web/dist was built (npm run build) — otherwise the backend only serves /api/*.
• Documents stuck on "processing"? Check the Embedding endpoint is reachable (KB_EMBEDDING_BASE_URL).
• Answers missing citations? The document must be ingested successfully and the question must relate to its content; the model will say so explicitly when it cannot answer from the knowledge base.

License

MIT