PocketDocs

30-second pitch: PocketDocs is a local-first RAG document Q&A system — sign up, upload any PDFs (contracts, papers, manuals, notes, anything), organize them into named collections, ask questions in a chat UI, and get answers grounded in your documents with inline [filename, p.N] (or pp.N-M for a chunk spanning pages) citations. Retrieval is hybrid (FAISS semantic + BM25 keyword search) and cross-encoder re-ranked before a Claude call generates the final answer, so answers stay traceable to a specific page instead of being a black box. Every account gets its own fully isolated documents.

Not to be confused with Async-Document-Processing-Workflow-System — a separate project also built around document upload, but a different problem entirely. That one is a document-processing pipeline: upload → background job → live progress tracking → review/edit extracted output → export. It's about orchestrating async work reliably (TypeScript, Docker Compose), and there's no retrieval, ranking, or Q&A involved. PocketDocs is about asking questions over documents you've already got — the hard problems here are retrieval quality (hybrid search, re-ranking) and grounding an LLM's answer in the right source, not job orchestration.

Problem

Generic chatbots hallucinate when asked about a specific PDF, and don't tell you which page a claim came from. Anyone reviewing a contract, paper, or spec needs an answer they can verify, not just an answer.

Approach

PDF upload -> continuous-stream text extraction (pypdf) -> token-window chunking (LangChain)
           -> embed (sentence-transformers, local) -> FAISS index (per-user, per-collection)
Question   -> hybrid search (weighted FAISS + BM25) -> optional cross-encoder re-rank (top 20 -> top 5)
           -> Claude, grounded on the top-5 chunks, forced to cite [source, p.N or pp.N-M]

• Ingestion (app/ingestion.py): the whole PDF is chunked as one continuous stream (500-token window, 50-token overlap), not per page, so a chunk can legitimately span a page boundary and keep a sentence that was split across it. Each chunk's character offsets are compared against every page's offsets to compute exactly which page(s) it touches, stored as metadata["pages"] — usually one page, two (or more) if it straddles a boundary.
• Storage (app/store.py): one FAISS index per authenticated user holds every chunk from every one of that user's collections, persisted under data/index/<user>/ so the backend survives a restart without re-ingesting. Collections aren't separate indexes — every chunk just carries a collection metadata field (a user-chosen name, e.g. a course or subject) — but every read path (retrieval, document listing, deletion, the graph view) filters by it, so a document uploaded into one collection never surfaces when a question is asked against another.
• Hybrid retrieval (app/retrieval.py): scoped to one collection first, then FAISS and BM25 scores (BM25 rebuilt fresh per query from that collection's chunks) are min-max normalized and combined as 0.6 * semantic + 0.4 * keyword (weights are constants in app/config.py).
• Re-ranking (app/reranker.py): the top 20 hybrid candidates are scored by a cross-encoder (ms-marco-MiniLM-L-6-v2) against the raw question and cut to the top 5 — a real relevance model, not just a fusion heuristic. Scores are cached per (query, chunk_id) (a bounded LRU, app/reranker.py), so a repeated question — or a new question whose hybrid candidates overlap a previous one's — skips re-scoring those pairs; a cold cache costs exactly what it always did, since misses are still batched into one predict() call. Re-ranking is also optional per-request (use_reranker on /chat and a "Precise mode" toggle in the UI) — see Results for what turning it off actually costs.
• Generation (app/generation.py): the top-5 chunks are passed to Claude with a system prompt that requires an inline citation copied verbatim from each chunk's [filename, ...] header — either p.N or pp.N-M — and instructs it to say "I don't have enough information" instead of guessing when the context doesn't answer the question.
• Collections (app/store.py, app/main.py): every upload, chat, document listing, and graph view takes a collection name (default "default" if you don't care) — any grouping you want, e.g. "Taxes 2025" and "Client Contracts" stay fully separate corpora sharing one FAISS index and one chunk_id space, so a question asked in one collection can never retrieve a chunk from another.
• Auth (app/auth.py): email/password signup and login, PBKDF2-HMAC-SHA256 password hashing with a random per-user salt (stdlib hashlib, no new dependency), opaque session tokens. Every document-touching endpoint requires a valid token and operates on that user's own DocumentStore and upload directory — this is what replaced the single global store with real per-user isolation. See Design Decisions for exactly how far this auth implementation does (and doesn't) go.
• API (app/main.py, FastAPI): POST /auth/signup, POST /auth/login, POST /auth/logout, POST /upload (PDFs + collection form field in, chunk counts out), POST /chat (question + collection + use_reranker in, {answer, citations} out), GET /collections (names + chunk counts), GET /documents?collection= and DELETE /documents/{filename}?collection= (list/remove an indexed PDF within a collection), GET /graph?collection= (that collection's document/chunk structure as a Graphviz DOT string), GET /health. Every endpoint except /health and /auth/* requires an Authorization: Bearer <token> header.
• UI (ui/streamlit_app.py): a login/signup gate before anything else. Once in, a collection selector (pick an existing one or type a new name) drives everything below it — upload, chat, indexed-document list, and knowledge graph are all scoped to whichever collection is active, and switching collections starts a fresh chat. Chat history, citations rendered as expandable sections showing the actual retrieved chunk text, a "Precise mode" toggle for the re-ranking fast path, and a "Knowledge graph" expander rendering each document's chunk structure (via st.graphviz_chart, no extra dependency needed — it renders a raw DOT string client-side). The graph is structural (doc → page chunks), not an extracted entity/relation graph — deleting a document removes its node and all child chunk nodes from both the graph and the retrieval index.

Design Decisions

Places the spec left open, and what was chosen:

• Continuous-stream chunking with page-range metadata, not per-page chunking. An earlier version of this project chunked each page independently to guarantee one unambiguous page number per chunk, at the explicit cost of losing context for sentences that straddle a page break. That tradeoff has been fixed: chunking now runs across the whole document, and a chunk's page(s) are computed from character offsets instead of assumed from which page loop iteration produced it (see app/ingestion.py). The real cost this revealed: on a small document, a chunk spanning 2-3 pages combined with retrieving several chunks means "top-5 chunks" can touch most of a 10-page document regardless of relevance ranking — see the honest caveat in Results. On a realistically sized document (hundreds of chunks), 5 chunks is a small fraction and this effect disappears; it's an artifact of the eval corpus being small, not of the chunking logic.
• Token counting uses LangChain's TokenTextSplitter (tiktoken-backed) rather than a word-count approximation, so the "500 tokens" in the spec means actual LLM tokens.
• Full-corpus semantic re-scoring: hybrid search asks FAISS for a similarity score against every indexed chunk (not just its own top-k) before fusing with BM25, so the two signals are compared on the same set. This is only fine at portfolio scale (single process, one FAISS index per user); a much larger corpus would need to fuse only each retriever's own top-k instead of rescoring everything on each query.
• Sample eval document is synthetic, not a fetched Wikipedia PDF. A generated 10-page spec for a fictional caching system ("Aria Cache") gives one unambiguous, checkable fact per page, which makes hit-rate and expected-answer scoring exact — a real-world PDF with fuzzier phrasing would make expected answers harder to write and hit-rate noisier to grade. Each page carries several hundred words of plausible elaboration, not just one sentence, specifically because continuous chunking would otherwise collapse a too-short document into one or two giant chunks spanning the whole thing — an earlier draft of this document did exactly that, and it was caught by actually inspecting chunk output before trusting the eval numbers, not by assuming the numbers were fine. See eval/generate_sample_doc.py.
• Auth is demo-grade, not production-grade. Real email/password signup and login exist (PBKDF2-HMAC-SHA256, random per-user salt, no plaintext passwords) and every user's documents are genuinely isolated — but there's no email verification, no password reset, no rate limiting or lockout on repeated failed logins, and session tokens live in memory only, so a backend restart logs everyone out (accounts themselves persist to disk in data/users.json). Good enough to demonstrate real per-user isolation in a portfolio project; not a drop-in replacement for a hardened auth provider in a real product. One thing that is fixed at production quality: login() always runs a real PBKDF2 hash even when the email doesn't exist, so response timing can't be used to enumerate which emails have accounts — measured directly before the fix (a ~450x difference between a known vs. unknown email) and after (~1.0x).
• Malformed uploads and oversized files fail cleanly, not with a stack trace. A .pdf that pypdf can't actually parse returns a 400 instead of an unhandled 500 (verified with a real corrupted file before and after); uploads are streamed to disk in bounded chunks and rejected with a 413 past config.MAX_UPLOAD_BYTES (25MB) instead of writing an unbounded file first and checking after.
• Per-user DocumentStore instances are cache-bounded, not unbounded. Each one holds a full FAISS index in memory; app/main.py keeps at most config.MAX_CACHED_STORES (50) resident and evicts the least-recently-used one (saving it to disk first) when a new user shows up past that cap — idle users reload transparently from disk on their next request instead of the process's memory growing forever with every distinct user seen.
• Missing ANTHROPIC_API_KEY never blocks setup. Ingestion, retrieval, and re-ranking all work with zero API keys (everything runs local models). Only /chat's final generation step needs the key, and it fails with a clear 503 + message rather than a stack trace. The eval harness does the same: it always reports a real hit rate, and skips (rather than fakes) the answer-relevancy score if no key is present.

Tradeoffs: hybrid search + re-ranking (latency vs. accuracy)

This pipeline does three retrieval passes before Claude ever sees a token: FAISS search, BM25 search, then a cross-encoder pass over 20 candidates. Each stage adds latency to buy accuracy:

• BM25 catches what embeddings miss — exact identifiers, version numbers, config keys (eviction_policy, port 7070) that a semantic embedding can blur together with similar text. Cost: a second index to maintain and rebuild on every ingest.

• The cross-encoder re-rank is the most expensive step per query — it runs a full transformer forward pass over 20 (query, chunk) pairs, not just a vector lookup — but it's also what actually catches cases where hybrid fusion's linear score gets the ranking wrong. On a small corpus this is milliseconds; it's the first thing to cut if latency matters more than precision (e.g., dropping straight from hybrid top-5 to Claude, no re-ranking at all).

• Net effect, measured for real (not reasoned about — use_reranker makes both paths runnable, and eval/run_eval.py runs the same 20 questions through both):

  Config	Hit rate	Avg retrieval latency	Avg answer relevancy
  With cross-encoder re-rank	100.00%	715.0 ms	0.643
  Without re-rank (hybrid top-5 only)	85.00%	14.4 ms	0.581

  Re-ranking is ~50x slower per query and, on this eval set, turns 3 of 20 questions from a miss into a hit, plus a real relevancy bump. Whether that trade is worth it depends entirely on the traffic pattern: fine for a single interactive user, the first thing to cut or move behind a queue if serving many concurrent users at low latency budgets.

What's still open:

• Cross-encoder score cache is per-process and in-memory; it doesn't survive a restart or help a second backend instance. A shared cache (Redis, or just persisting it) would help a multi-instance deployment.
• Real production auth (email verification, password reset, rate limiting, persistent sessions) instead of the demo-grade implementation described in Design Decisions.
• A larger, non-synthetic eval corpus. The 10-page synthetic document is large enough to avoid collapsing into one giant chunk, but still small enough that "top-5 of ~10 chunks" touches most pages regardless of relevance — a real multi-hundred-chunk corpus wouldn't have this ceiling effect.

Results

Full per-question breakdown: eval/results.md (real numbers from an actual run against the synthetic Aria Cache spec, 20 question/answer pairs, re-ranking on and off).

• Hit rate with re-ranking: 100.00% (20/20) — every question retrieved its expected page somewhere in its top-5 chunks. Without re-ranking: 85.00% (17/20) — see the Tradeoffs section above for the full latency/accuracy comparison.
• Avg answer relevancy: 0.643 with re-ranking, 0.581 without (cosine similarity between each generated answer and its expected answer, embedded with the same sentence-transformers model used for retrieval). Per-question scores range from 0.48 to 0.91 in eval/results.md; variance largely tracks how much a question's expected answer overlaps in wording with Claude's citation-heavy phrasing, not retrieval failures.

An honest caveat about this number, found by inspecting the actual retrieved pages, not just the pass/fail column: because chunks can now span 2-3 pages each (see Design Decisions), and the synthetic document only has ~10 chunks total, the top-5 chunks for a typical question touch 7-10 of the document's 10 pages — the hit-rate ceiling is real but weaker evidence of precise per-question discrimination than it looks. The with/without re-ranking delta above is still a valid comparison (everything else held constant), but absolute hit-rate on this small corpus should be read as "the pipeline works end-to-end," not "this precisely nails down the one relevant page." A real, much larger document wouldn't have this ceiling effect, since 5 chunks out of hundreds is a small, targeted fraction.

How to run

Requires Python 3.11+.

git clone https://github.com/HarshitaSobhani/PocketDocs.git
cd PocketDocs
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# edit .env and set ANTHROPIC_API_KEY (optional — the app runs without it, see above)

# Terminal 1: backend
uvicorn app.main:app --reload --port 8000

# Terminal 2: frontend
streamlit run ui/streamlit_app.py

Open the Streamlit URL it prints. First time here, use the Sign up tab with any email and an 8+ character password — there's no email verification, so anything shaped like an email address works. Then upload a PDF in the sidebar, click Add to pocket, and ask a question in the chat box.

Run tests:

pytest

Run the eval harness (generates the synthetic sample PDF on first run):

python eval/run_eval.py

Structure

/app      FastAPI backend: auth, ingestion, hybrid retrieval, re-ranking, generation
/ui       Streamlit chat frontend (login/signup gate + chat)
/eval     Synthetic sample PDF, eval set, scoring script, results.md
/tests    pytest — chunking, retrieval, API, auth