A locally-hosted Hybrid RAG pipeline for clinical genomic queries
Combining air-gapped local databases · Live expert APIs · Local LLM inference · Zero hallucination tolerance
- What It Does
- System at a Glance
- Data Sources
- The 6-Node LangGraph Pipeline
- Evaluation Framework
- Offline Ingestion Pipelines
- Database Schema
- Tech Stack
- Project Structure
- Setup & Installation
- Configuration
- Example Query & Response
- Roadmap
This system answers complex clinical and genomic queries — variant classification, NCCN protocol retrieval, and ClinGen expert panel validation — with zero (Almost) hallucinations and full data privacy.
Example query:
"What is the clinical significance of rs879254116 in BRCA1 and what cancer screening protocol should the patient follow? Also confirm the ClinGen expert panel validity for BRCA1."
Example output structure:
**Clinical Summary**
Variant rs879254116 in BRCA1 is classified as Pathogenic.
[Source: Clinvar, Reference: rs879254116]
**ClinGen Expert Panel Validity**
* BRCA1 has gene-disease validity curated by ClinGen expert panel.
Actionability curations available. Last curated: 08/29/2024.
[Source: ClinGen, Reference: BRCA1 (HGNC:1100)]
**Cancer Screening Protocol**
* Annual breast MRI and mammography starting at age 25-29 years.
[Source: Genetic-Familial High-Risk Assessment, Reference: NCCN Guidelines
Version 3.2026 Genetic/Familial High-Risk Assessment...]
* Risk-reducing salpingo-oophorectomy between ages 35-40.
[Source: Genetic-Familial High-Risk Assessment,
Reference: Bilateral Salpingo-Oophorectomy]
Key guarantees:
| Guarantee | How it's enforced |
|---|---|
| 🔒 No patient data leaves the system | All LLM inference via local Ollama; gnomAD receives only positional coords |
| 📎 Every claim has a citation | Citation manifest injected into prompt; enforcer post-validates all refs |
| 🚫 No hallucinated variant biology | Rule 5 forbids inventing variant types or molecular mechanisms from memory |
| 🧬 No cross-gene table slippage | Rule 6 requires gene-name verification in every NCCN table row |
| ✅ Structured, auditable output | 3-section Pydantic schema: Summary · ClinGen · Screening |
| ⚡ Safe failure | Insufficient data → predefined refusal, never a guess |
┌─────────────────────────────────────────────────────────────────────────────┐
│ PRIVACY-PRESERVING CDSS — FULL SYSTEM │
└─────────────────────────────────────────────────────────────────────────────┘
OFFLINE (run once) ONLINE (per query)
───────────────── ────────────────────────────────────
ClinVar FTP (~250MB) 👨⚕️ Clinician
│ │ POST /query
▼ ▼
clinvar_ingestion.py ┌─── FastAPI (port 5656) ───────────────┐
• Download + MD5 verify │ │
• Filter: GRCh38, P/LP/B/LB │ LangGraph 6-Node Pipeline │
• Batch INSERT (1000/tx) │ │
│ │ ① Decomposer │
▼ │ ↓ fan-out (parallel) │
┌──────────────┐ indexing.py │ ② DB_Retriever ── ③ PDF_Retriever │
│ PostgreSQL │◄── Docling │ ↓ merge ↓ │
│ + pgvector │ Docling │ ④ Evaluator ◄────────────────────── │
│ │ │ ↓ │
│ variants │ 800-char │ ⑤ Generator │
│ table │ chunks │ ↓ │
│ │ │ ⑥ Citation_Enforcer │
│ medical_ │ │ ↓ │
│ documents │ │ QueryResponse (JSON) │
│ (768-dim) │ └───────────────────────────────────────┘
└─────────────────────────────────────────────┘
▲ │
│ Live calls (per query) ▼
┌──────┴───────────────────────┐ answer + citations + confidence
│ gnomAD v4 │ ClinGen REST │
│ (allele │ (gene expert │
│ frequency) │ panels) │
└──────────────────────────────┘
┌────────────────────────────────────────────────┐
│ FOUR DATA SOURCES EXPLAINED │
└────────────────────────────────────────────────┘
┌──────────────────────┐ ┌──────────────────────┐
│ 🗄️ PostgreSQL │ │ 🔮 pgvector │
│ ClinVar (local) │ │ NCCN (local) │
│ │ │ │
│ • Millions of │ │ • NCCN Breast v2026 │
│ variant records │ │ • NCCN Genetic/ │
│ • rsID lookups │ │ Familial High-Risk│
│ • Gene-level scan │ │ • 768-dim BGE embed │
│ • SQL precision │ │ embeddings │
│ │ │ │
│ ✅ Air-gapped │ │ ✅ Air-gapped │
│ ✅ Sub-millisecond │ │ ✅ Semantic search │
│ ✅ Exact match │ └─────────────────────┘
└──────────────────────┘
┌──────────────────────┐ ┌──────────────────────┐
│ 🌐 gnomAD v4 API │ │ 🌐 ClinGen REST API │
│ (live, opt-out) │ │ (live, gene-only) │
│ │ │ │
│ • Population allele │ │ • Gene-disease │
│ frequency │ │ validity │
│ • BA1 rule: │ │ • Actionability │
│ AF ≥ 5% → Benign │ │ • Dosage sensitivity│
│ • PM2 flag: │ │ • Variant expert │
│ absent → noted │ │ panel status │
│ • Sends: chrom-pos- │ │ • Last curated date │
│ ref-alt only │ │ • Sends: gene symbol│
│ │ │ only (e.g. BRCA1) │
│ ⚠️ Disable with │ │ │
│ ENABLE_GNOMAD=false │ │ ✅ Real-time data │
└──────────────────────┘ └──────────────────────┘
┌─────────────────────────┐
│ CDSSGraphState (shared │
│ across all 6 nodes) │
│ │
│ query: str │
│ gene: Optional[str] │
│ sub_queries: list │
│ trusted_chunks: list │
│ candidate_chunks: list │
│ verified_chunks: list │
│ draft_answer: str │
│ final_answer: str │
│ citations: list │
│ confidence: str │
└─────────────────────────┘
START
│
▼
╔═══════════╗
║ Node 1 ║ decompose_node()
║ Decomposer║ • Extract gene symbol via regex
╚═════╤═════╝ • Keyword match → typed SubQuery list
│ • Writes: gene, sub_queries
│
├──────────────────────────┐
│ (parallel via Send API) │
▼ ▼
╔═══════════╗ ╔═════════════╗
║ Node 2 ║ ║ Node 3 ║
║DB_Retriever ║PDF_Retriever║
╚═════╤═════╝ ╚══════╤══════╝
│ │
ClinVar SQL Multi-query
gnomAD API expansion (LLM)
ClinGen API pgvector search
│ Deduplication
│ │
└───────────┬─────────────┘
│ (merge via Annotated[list, operator.add])
▼
╔══════════════╗
║ Node 4 ║ evaluate_node()
║ Evaluator ║ • Deduplicate candidate_chunks
╚══════╤═══════╝ • BGE cross-encoder rerank
│ • CRAG grade (CORRECT/AMBIGUOUS/INCORRECT)
│ • Gene-filter NCCN chunks
│ • Cap 6 chunks per source
│ • Merge: trusted + filtered_candidates
▼
╔══════════════╗
║ Node 5 ║ generate_node()
║ Generator ║ • Build ⚑ VARIANT FACTS context block
╚══════╤═══════╝ • Build numbered CITATION MANIFEST
│ • JSON-schema constrained Ollama call
│ • Parse ClinicalResponse → markdown
▼
╔══════════════════╗
║ Node 6 ║ citation_node()
║Citation_Enforcer ║ • fix_hallucinated_citations()
╚══════╤═══════╝ • extract_citations()
│ • Confidence scoring
▼
END → QueryResponse{answer, citations, confidence}
Breaks the query into typed SubQuery objects via keyword and regex matching. Critical design decision: each sub-query gets a focused text containing only the keywords relevant to that topic — NOT the full user query. This prevents embedding dilution and ensures the reranker scores chunks against the correct topic.
Query: "What is rs879254116 in BRCA1 and what NCCN screening applies? ClinGen validity?"
│
┌────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌──────────────────────┐ ┌────────────────────┐
│SubQuery 1 │ │SubQuery 2 │ │SubQuery 3 │
│target: postgres │ │target: vector_db │ │target: clingen │
│type: data_ext. │ │type: screening_ret. │ │type: clingen_lookup│
│ │ │ │ │ │
│text: "Get │ │text: "NCCN cancer │ │text: original query│
│clinical signif. │ │screening surveillance│ │(API call, not │
│for rs879254116" │ │mammography MRI BRCA1 │ │ vector search) │
│ │ │carrier management" │ │ │
│✅ Already focused│ │✅ Focused — with │ │✅ API-based │
└─────────────────┘ │ only NCCN terms │ └────────────────────┘
└──────────────────────┘
Why focused sub-query text matters: When the reranker sees focused topic text vs a relevant document chunk, it scores much HIGHER than when using the full mixed query. Focused text dramatically improves relevance scoring.
Keyword groups that trigger each sub-query type:
| Detected pattern | Sub-query target | Sub-query type | Focused text template |
|---|---|---|---|
rs\d+, NM_\d+.\d+, NP_\d+.\d+ |
postgres |
data_extraction |
"Get clinical significance for {rsID}" |
nccn, screening, surveillance, rrso, hereditary, mammography, mri |
vector_db |
screening_retrieval |
"NCCN cancer screening surveillance mammography MRI risk-reducing {gene} carrier" |
chemotherapy, neoadjuvant, radiotherapy, treatment regimen |
vector_db |
protocol_retrieval |
"Cancer treatment protocol chemotherapy surgery radiation {gene}" |
clingen, gene validity, expert panel, actionability |
clingen |
clingen_lookup |
Original query (ClinGen is API-based, not vector search) |
| (no match) | vector_db |
general |
Original query (fallback, no category filter) |
Screening takes priority over protocol: if both keywords appear (e.g. "BRCA1 screening protocol"), only
screening_retrievalfires —protocol_retrievalis suppressed to avoid noise.
Fetches verified structured facts from three sources. Results become trusted_chunks — they bypass CRAG filtering and go directly to the LLM as ground truth.
For each SubQuery where target == "postgres":
┌─────────────────────────────────────────────────────────────────┐
│ rsID found in query? │
│ YES NO (gene only) │
│ │ │ │
│ ▼ ▼ │
│ get_variant_by_rsid(rsid) get_variant_by_gene(gene) │
│ │ │ │
│ ▼ ▼ │
│ gnomAD GraphQL API Multiple RetrievedChunks │
│ chrom-pos-ref-alt query (source=Clinvar) │
│ │ │
│ Found? Not found? │
│ │ │ │
│ ▼ ▼ │
│ AF + BA1 PM2 may apply note │
│ (source=gnomAD) │
└─────────────────────────────────────────────────────────────────┘
For each SubQuery where target == "clingen":
┌─────────────────────────────────────────────────────────────────┐
│ GET /api/genes?search={gene_symbol} │
│ │ │
│ ▼ │
│ Filter to exact symbol match (BRCA1 ≠ BRCA1P1) │
│ │ │
│ ▼ │
│ RetrievedChunk(source=ClinGen): │
│ • Gene-disease validity: YES/NOT CURATED │
│ • Actionability curations: YES/NO │
│ • Dosage sensitivity: Curated/Not curated │
│ • Variant expert panel: active/not active │
│ • date_last_curated │
└─────────────────────────────────────────────────────────────────┘
Searches the local pgvector database using multi-query expansion. Each sub-query's results are reranked per-subquery against their own focused text before being pooled — ensuring screening chunks are scored against screening queries, not against the full multi-topic user query.
For each SubQuery where target == "vector_db":
STEP 1 — Query Expansion (Ollama LLM, JSON-schema constrained)
┌───────────────────────────────────────────────────────────────┐
│ seed = focused sub-query text (from Decomposer) │
│ │
│ e.g. "NCCN cancer screening surveillance BRCA1" │
│ → LLM generates 3 variants: │
│ • BRCA1 carrier annual breast MRI mammography age │
│ • risk-reducing salpingo-oophorectomy RRSO age BRCA │
│ • hereditary breast ovarian cancer surveillance protocol │
│ │
│ Output: ExpandedQueries {queries: [q1, q2, q3]} │
└───────────────────────────────────────────────────────────────┘
STEP 2 — Vector Search (4 queries × top_k=15)
┌───────────────────────────────────────────────────────────────┐
│ [focused_sub_query, q1, q2, q3] │
│ │ │
│ ▼ for each query: │
│ embed_text() → 768-dim BGE-base-en-v1.5 vector │
│ │ │
│ ▼ │
│ SELECT ... FROM medical_documents │
│ WHERE category = {protocol|screening_protocol} │
│ ORDER BY embedding <=> query_vector │
│ LIMIT 15 │
└───────────────────────────────────────────────────────────────┘
STEP 3 — Deduplication + Per-Subquery Reranking
┌───────────────────────────────────────────────────────────────┐
│ 4 queries × 15 results → deduplicate → ~30-40 unique chunks │
│ │
│ ★ BGE cross-encoder reranks THIS batch against the focused │
│ sub-query text (not the full user query) │
│ │
│ Chunks arrive at Evaluator already scored per-topic │
└───────────────────────────────────────────────────────────────┘
Indexed documents:
| File | Source name | Category | Parser |
|---|---|---|---|
Breast.pdf |
NCCN_Breast_v2_2026 |
protocol |
Docling (table-aware) |
Genetic-Familial High-Risk Assessment...pdf |
Genetic-Familial High-Risk Assessment |
screening_protocol |
Docling (table-aware) |
Merges both retrieval branches, applies quality filtering, and produces the final verified_chunks list that gets sent to the LLM. Chunks arrive pre-scored from per-subquery reranking in PDF_Retriever — no re-reranking against the full query happens here.
trusted_chunks (DB_Retriever) candidate_chunks (PDF_Retriever)
│ (already scored per-subquery)
│ │
│ ▼
│ STEP 1: Deduplicate by text[:200]
│ (same chunk from multiple sub-queries
│ → keep first occurrence's score)
│ │
│ ▼
│ STEP 2: Sort by score (descending)
│ (chunks pre-scored in PDF_Retriever;
│ NO re-reranking against full query)
│ │
│ ▼
│ STEP 3: CRAG Grading
│ ┌─────────────────────────────────┐
│ │ score ≥ 0.05 → CORRECT ✅ │
│ │ score 0.01–0.05 → AMBIGUOUS ⚠️ │
│ │ score < 0.01 → INCORRECT ❌ │
│ │ (dropped) │
│ └─────────────────────────────────┘
│ │
│ STEP 4: Gene Filter (NCCN only)
│ ┌─────────────────────────────────┐
│ │ NCCN chunk doesn't mention │
│ │ target gene → DROPPED │
│ │ (prevents STK11/CDH1 protocols │
│ │ appearing in BRCA1 queries) │
│ └─────────────────────────────────┘
│ │
│ STEP 5: Source cap (max 6 per source)
│ │
└──────────────────┬─────────────────────┘
▼
verified_chunks
(trusted DB chunks + filtered PDF chunks)
typically 3 DB + 8–12 PDF chunks
Previous design flaw (fixed): The Evaluator previously re-reranked ALL chunks against the full multi-topic user query. This caused chunks to score very low (mixing unrelated terms diluted the relevance signal) and get dropped as INCORRECT. Now chunks are scored per-subquery in PDF_Retriever before reaching the Evaluator.
Builds a structured clinical response using grammar-constrained JSON generation and a 6-rule system prompt.
verified_chunks
│
▼
build_system_prompt() — 6 anti-hallucination rules:
┌──────────────────────────────────────────────────────┐
│ RULE 1: Variant facts block is ground truth │
│ RULE 2: Citations must be copied from manifest │
│ RULE 3: Exact ages and terminology from guidelines │
│ RULE 4: If not in context, say "Data unavailable" │
│ RULE 5: Never invent variant biology (frameshift/ │
│ missense/de novo/computational predictions) │
│ RULE 6: Table row isolation — verify gene name in │
│ every NCCN table row before extracting data │
└──────────────────────────────────────────────────────┘
│
▼
build_context_block()
┌──────────────────────────────────────────────────────┐
│ ⚑ ═══════════════════════════════════════════════ │
│ ⚑ VARIANT DATABASE FACTS — READ THIS FIRST │
│ ⚑ ═══════════════════════════════════════════════ │
│ │
│ [Source: Clinvar, Reference: rs879254116] │
│ *** CONFIRMED CLASSIFICATION: Pathogenic *** │
│ │
│ [Source: gnomAD, Reference: rs879254116] │
│ Variant not found. Absence from controls noted. │
│ │
│ [Source: ClinGen, Reference: BRCA1 (HGNC:1100)] │
│ Gene-disease validity: True. Actionability: True. │
│ │
│ ── CLINICAL GUIDELINES ────────────────────────── │
│ [Source: Genetic-Familial High-Risk Assessment, │
│ Reference: BRCA PATHOGENIC VARIANT-POSITIVE MGMT] │
│ NCCN screening protocol text... │
│ ... │
└──────────────────────────────────────────────────────┘
│
▼
_build_reference_manifest()
┌──────────────────────────────────────────────────────┐
│ CITATION MANIFEST — ONLY these are permitted │
│ [1] [Source: Clinvar, Reference: rs879254116] │
│ [2] [Source: gnomAD, Reference: rs879254116] │
│ [3] [Source: ClinGen, Reference: BRCA1 (HGNC:1100)]│
│ [4] [Source: Genetic-Familial..., Reference: BRCA │
│ PATHOGENIC VARIANT-POSITIVE MANAGEMENT] │
└──────────────────────────────────────────────────────┘
│
▼
ollama.chat(format=ClinicalResponse.model_json_schema())
┌──────────────────────────────────────────────────────┐
│ Grammar-constrained — model CANNOT output free text │
│ │
│ ClinicalResponse { │
│ summary: ClinicalClaim, ← variant facts │
│ clingen_validity: [ClinicalClaim], ← ClinGen data │
│ screening_protocol: [ClinicalClaim] ← NCCN │
│ } │
│ │
│ ClinicalClaim { │
│ text: str, ← full sentence explanation │
│ citations: [str] ← copied from manifest ONLY │
│ } │
└──────────────────────────────────────────────────────┘
│
▼
_strip_thinking() → remove <think>...</think> blocks
│
▼
Parse JSON → render markdown sections → draft_answer
Post-processes the final answer, fixes hallucinated citations, and computes the confidence score.
draft_answer + verified_chunks
│
├─────────────────────────────────────────┐
▼ ▼
fix_hallucinated_citations() extract_citations()
┌──────────────────────────┐ ┌──────────────────────┐
│ For each [Source: X, │ │ Regex scan for all │
│ Reference: Y] in text: │ │ [Source:...] tags │
│ │ │ Deduplicate │
│ Source in chunks? │ │ → citations list │
│ ├─YES: ref valid? ──────┤ └──────────────────────┘
│ │ ├─YES: leave alone │
│ │ └─NO: remap to │
│ │ first valid ref │ Confidence scoring:
│ └─NO: keyword-score │ ┌──────────────────────┐
│ 200-char context │ │ DB hits ≥ 2 │
│ ≥2 hits → remap │ │ AND PDF hits ≥ 1 │
│ to correct source │ │ → HIGH │
└──────────────────────────┘ │ │
│ DB hits OR PDF hits │
│ → MEDIUM │
│ │
│ Neither → LOW │
└──────────────────────┘
NCBI FTP Server
│
▼ urllib.request.urlretrieve()
variant_summary.txt.gz (~250MB)
│
▼ MD5 checksum verify (re-download if corrupted)
│
▼ gzip.open() + csv.DictReader (tab-separated)
│
▼ is_relevant() filter:
• Assembly == "GRCh38" (discard GRCh37)
• ClinicalSignificance contains: Pathogenic / Likely Pathogenic
/ Benign / Likely Benign
• ReviewStatus ≠ "no assertion" / "no interpretation"
• rsID present (not "-1")
│
▼ Batch INSERT (1000 rows per transaction)
ON CONFLICT (rsid) DO UPDATE ← keeps database current
│
▼ PostgreSQL variants table
docs/manifest.json
(source, category, gene, parser per file)
│
▼ discover_documents()
│
├─── Breast.pdf (NCCN) ──► parser: docling
│ │
│ ▼ Docling (do_table_structure=True, do_ocr=False)
│ │ Batch 10 pages at a time → Markdown
│ │ Scrub: NCCN copyright, version headers, ToC
│ │
│ ▼ table_aware_split():
│ MarkdownHeaderSplitter (H1/H2/H3)
│ + isolate pipe-table blocks separately
│
└─── Genetic-Familial...pdf ──► parser: docling (same path)
│
▼ (same as above)
All paths converge:
│
▼ RecursiveCharacterTextSplitter
chunk_size=800, chunk_overlap=100
enriched_child = "[Header_2: SECTION NAME]\nchunk text"
│
▼ embed_text() → BGE-base-en-v1.5 768-dim vector
│
▼ INSERT INTO medical_documents
(source, category, gene, chunk_text, metadata,
parent_text, embedding::vector)
Three-phase evaluation pipeline for dissertation-grade empirical evidence:
Phase 1 (run_evaluation.py) Phase 2 (run_ragas.py) Phase 3 (run_ablation.py)
────────────────────────── ────────────────────────── ──────────────────────────
API server (POST /query/detailed) Reads raw_outputs_*.json Runs pipeline with mock patches
│ │ │
▼ ▼ ▼
28 golden test cases Local LLM-as-judge 3 configs:
• Hallucination rate (draft) (ministral-3:14b via Ollama) • Full pipeline
• Guardrail effectiveness │ • No CRAG filter
• Citation accuracy │ • No guardrails
• Screening keyword accuracy ▼
│ 3 Ragas metrics:
▼ • Faithfulness
raw_outputs_*.json • Response Relevancy
run_*.md + charts/ • Context Precision
| Category | Example Cases | What’s Tested |
|---|---|---|
| Multi-source | TC-001 (ClinVar + gnomAD + ClinGen + NCCN) | Full pipeline integration |
| Cross-gene traps | TC-006, TC-013 (BRCA1 + colonoscopy) | Rule 6 table row isolation |
| Safe failures | TC-004 (non-existent variant) | Refusal without hallucination |
| gnomAD guardrails | TC-005, TC-010 (AF present / absent) | BA1/PM2 accuracy |
| Moderate-penetrance | TC-026 (PALB2), TC-027 (CHEK2), TC-028 (ATM) | Gene-specific protocols |
Phase 1 — Hallucination & Citation (28 cases):
| Metric | Result |
|---|---|
| Pass rate | 27/28 (96.4%) |
| Avg hallucination (draft) | 1.2% |
| Citation accuracy | 100.0% |
| Keyword accuracy | 89.7% |
Phase 2 — Ragas LLM-as-Judge (27 cases, baseline run):
| Metric | Score | Notes |
|---|---|---|
| Faithfulness | 0.466 | Baseline — 6 N/A cases due to token limits; re-run after pipeline fixes for accurate score |
| Response Relevancy | 0.775 | ✅ Answers on-topic |
| Context Precision | 0.632 | Bibliography chunks inflated CRAG scores; fixed in reranker |
Ragas judge:
ministral-3:14bvia Ollama OpenAI-compatible endpoint (/v1). Embeddings:BAAI/bge-base-en-v1.5(local sentence-transformers). No external API calls during evaluation.
# Phase 1 — API server must be running
python evaluation/run_evaluation.py
# Phase 2 — Ragas (reads most recent Phase 1 raw_outputs_*.json automatically)
python -m evaluation.run_ragas
# Phase 3 — Ablation study
python evaluation/run_ablation.py┌─────────────────────────────────────────────────────────────────┐
│ variants │
├──────────────────────┬──────────────────────────────────────────┤
│ id SERIAL PK │ rsid VARCHAR(20) UNIQUE │
│ gene_symbol TEXT │ chromosome VARCHAR(10) │
│ position BIGINT │ ref_allele TEXT │
│ alt_allele TEXT │ clinical_significance TEXT │
│ review_status TEXT │ condition TEXT │
│ last_evaluated DATE │ created_at TIMESTAMP │
├──────────────────────┴──────────────────────────────────────────┤
│ Indexes: idx_variants_rsid (primary lookup) │
│ idx_variants_gene (gene-level scans) │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ medical_documents │
├──────────────────────┬──────────────────────────────────────────┤
│ id SERIAL PK │ source VARCHAR(50) │
│ category VARCHAR(50)│ gene VARCHAR(50) │
│ chunk_text TEXT │ embedding vector(768) ← pgvector │
│ metadata JSONB │ parent_text TEXT │
│ created_at TIMESTAMP│ │
├──────────────────────┴──────────────────────────────────────────┤
│ metadata JSONB shape: │
│ { "Header_1": "...", "Header_2": "SECTION NAME", │
│ "Header_3": "...", "page": 33 } │
│ │
│ Reference extraction priority: │
│ Header_6 → Header_5 → ... → Header_1 → page number │
│ │
│ Indexes: idx_documents_source │
│ idx_documents_embedding (ivfflat cosine_ops) │
│ idx_documents_parent_text │
└─────────────────────────────────────────────────────────────────┘
| Layer | Technology | Version | Role |
|---|---|---|---|
| API | FastAPI + Uvicorn | 0.131 / 0.41 | HTTP interface, Pydantic I/O validation |
| Orchestration | LangGraph (StateGraph) | 1.0.10 | 6-node DAG, parallel Send-based fan-out |
| LLM | Ollama | 0.6.1 | Local inference — query expansion, generation |
| Embeddings | SentenceTransformer | 5.2.3 | BGE-base-en-v1.5 768-dim vectors |
| Reranking | BAAI/bge-reranker-large | — | Cross-encoder relevance scoring |
| Database | PostgreSQL 17 + pgvector | psycopg2 2.9.11 | Structured variants + vector similarity search |
| Connection pool | psycopg2 ThreadedConnectionPool | — | min=2 / max=10 connections shared |
| Docling | 2.77.0 | Table-structure-aware PDF → Markdown | |
| Chunking | LangChain text splitters | 1.2.10 | MarkdownHeader + Recursive (800 chars, 100 overlap) |
| External API 1 | gnomAD v4 GraphQL | httpx 0.28.1 | Allele frequency, BA1/PM2 annotations |
| External API 2 | ClinGen REST | httpx 0.28.1 | Gene validity, expert panel curation |
| Config | pydantic-settings | 2.13.1 | .env-based configuration |
| Container | Docker Compose | — | pgvector/pgAdmin |
| Evaluation | Ragas + langchain-huggingface | — | LLM-as-judge: Faithfulness, Relevancy, Context Precision |
Privacy-Preserving-CDSS/
│
├── app/
│ ├── main.py FastAPI entrypoint — port 5656
│ ├── config.py Pydantic settings, reads .env
│ │
│ ├── api/
│ │ ├── router.py POST /query → LangGraph invoke
│ │ └── schemas.py QueryRequest, QueryResponse, Citation
│ │
│ ├── db/
│ │ ├── pool.py ★ Shared ThreadedConnectionPool (min=2, max=10)
│ │ ├── postgres/
│ │ │ ├── clinvar_schema.sql DDL: variants + medical_documents + indexes
│ │ │ └── clinvar_ingestion.py Download, verify MD5, batch ingest ClinVar
│ │ └── vector/
│ │ └── indexing.py PDF→chunk→embed→pgvector pipeline
│ │
│ ├── models/
│ │ └── embeddings.py SentenceTransformer wrapper (BGE-base-en-v1.5)
│ │
│ └── pipeline/
│ ├── decomposition.py Keyword routing → SubQuery list
│ │
│ ├── construction/
│ │ └── _deprecated/ Deprecated stubs (self_query, text_to_sql)
│ │
│ ├── graph/
│ │ ├── state.py CDSSGraphState TypedDict
│ │ ├── nodes.py 6 node functions
│ │ └── workflow.py LangGraph DAG compilation
│ │
│ ├── retrieval/
│ │ ├── multi_query.py LLM expansion + pgvector search + dedup
│ │ ├── reranker.py BGE cross-encoder + RetrievedChunk factories
│ │ └── crag_evaluator.py Score-based chunk grading (0.05 / 0.01 thresholds)
│ │
│ ├── generation/
│ │ ├── guardrails.py System prompt + ⚑ context block builder
│ │ ├── self_rag.py JSON-schema gen + manifest + thinking strip
│ │ └── citation_enforcer.py Hallucination fix + citation extraction
│ │
│ └── sources/
│ ├── postgres_client.py ClinVar: get_variant_by_rsid / get_variant_by_gene
│ ├── vector_client.py pgvector cosine search (named params, pooled)
│ ├── gnomad_client.py gnomAD GraphQL (allele freq, BA1/PM2)
│ └── clingen_client.py ClinGen REST + gene symbol extraction
│
├── docs/
│ ├── manifest.json Parser + source + category metadata per PDF
│ ├── protocols/ NCCN Breast v2 2026
│ └── screening/ NCCN Genetic/Familial High-Risk Assessment
│
├── evaluation/
│ ├── golden_set.json 28 clinical test cases
│ ├── run_evaluation.py Phase 1 — hallucination + citation scoring
│ ├── run_ragas.py Phase 2 — Ragas LLM-as-judge (Faithfulness, etc.)
│ ├── run_ablation.py Phase 3 — ablation study (Full vs No-CRAG vs No-Guardrails)
│ └── results/ Timestamped .md reports, charts, raw JSON
│
├── tests/
│ ├── test_routing.py decompose_query() unit tests
│ ├── test_retrieval.py reranker + CRAG unit tests
│ ├── test_generation.py citation enforcer unit tests
│ └── test_clingen_client.py gene extraction unit tests
│
├── docker-compose.yml pgvector (pg17) + pgAdmin
├── .env.example
└── requirements.txt
| Requirement | Version | Purpose |
|---|---|---|
| Python | 3.10+ | Runtime |
| Docker Desktop | latest | PostgreSQL + pgvector |
| Ollama | latest | Local LLM inference |
1. Clone
git clone https://github.com/RenX86/Privacy-Preserving-CDSS.git
cd Privacy-Preserving-CDSS2. Install dependencies
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt3. Pull a local LLM
# Best quality for this task (recommended)
ollama pull ministral-3:14b
# Alternatives — smaller but faster
ollama pull llama3.1:8b
ollama pull qwen3.5:9b4. Configure environment
cp .env.example .env
# Open .env and set LOCAL_LLM_MODEL to your chosen model5. Start PostgreSQL + pgvector
docker compose up -d postgres
# Wait ~10s for healthcheck to pass6. Ingest ClinVar variants
# Downloads ~250MB from NCBI FTP, verifies MD5 checksum,
# filters to GRCh38 P/LP/B/LB variants, batch inserts into PostgreSQL
python -m app.db.postgres.clinvar_ingestion7. Index medical documents
# Reads docs/ folder, routes each PDF to correct parser,
# chunks → embeds → inserts into pgvector
python app/db/vector/indexing.py8. Start the API
python -m uvicorn app.main:app --port 5656 --reload
# Open http://localhost:5656/docs for Swagger UI9. Run a query
# Simple response (answer + citations + confidence)
curl -X POST http://localhost:5656/query \
-H "Content-Type: application/json" \
-d '{
"query": "What is the clinical significance of rs879254116 in BRCA1 and what cancer screening protocol should the patient follow according to NCCN guidelines? Also confirm the ClinGen expert panel validity for BRCA1."
}'
# Detailed response (includes draft_answer, node trace, timing — used by evaluation)
curl -X POST http://localhost:5656/query/detailed \
-H "Content-Type: application/json" \
-d '{"query": "What is rs879254116 in BRCA1?"}'API endpoints:
| Endpoint | Response | Use case |
|---|---|---|
POST /query |
QueryResponse — answer, citations, confidence |
Production queries |
POST /query/detailed |
InstrumentedResponse — adds draft_answer, gene, trace, timing |
Evaluation, debugging |
GET /health |
{"status": "online"} |
Health checks |
# ── PostgreSQL ────────────────────────────────────────────────────
POSTGRES_USER=cdss-user
POSTGRES_PASSWORD=cdss_password
POSTGRES_DB=cdss_db
POSTGRES_PORT=5432
POSTGRES_URL=postgresql://cdss-user:cdss_password@localhost:5432/cdss_db
# ── Local LLM via Ollama ──────────────────────────────────────────
LOCAL_LLM_URL=http://localhost:11434
LOCAL_LLM_MODEL=ministral-3:14b # any ollama model — swap freely
# ── Embedding Model ───────────────────────────────────────────────
EMBEDDING_MODEL=BAAI/bge-base-en-v1.5
# ── ClinGen API ───────────────────────────────────────────────────
CLINGEN_API_URL=https://search.clinicalgenome.org/kb
# ── Privacy flags ─────────────────────────────────────────────────
# false = fully air-gapped, disables live gnomAD allele frequency calls
# true = enables gnomAD BA1/PM2 annotations (sends chrom-pos-ref-alt only)
ENABLE_GNOMAD_LOOKUP=trueLLM model selection guide:
| Model | Size | Speed | Quality | Notes |
|---|---|---|---|---|
ministral-3:14b |
9GB | ★★ | ★★★★★ | Best citation compliance + structured output (recommended) |
qwen3.5:9b |
6.6GB | ★★★ | ★★★★ | 256K context, good reasoning, requires Ollama ≥ v0.17 |
llama3.1:8b |
4.7GB | ★★★★ | ★★ | Faster but poor citation compliance and short output |
Live run log (ministral-3:14b) — with focused sub-queries and per-subquery reranking:
[STARTING LANGGRAPH] Query: What is the clinical significance of rs879254116
in BRCA1 and what cancer screening protocol should the patient follow
according to NCCN guidelines. Also confirm the ClinGen expert panel
validity for BRCA1.
[Decomposer] Screening sub-query: NCCN cancer screening surveillance
mammography MRI risk-reducing prophylactic salpingo-oophorectomy BRCA1...
PostgreSQL connection pool initialised (min=2, max=10)
[gnomAD] Querying variant 17-43049159-GA-G → Variant not found
[ClinGen] 1 record(s) for BRCA1
symbol=BRCA1 | validity=True | actionability=True | curated=08/29/2024
[MultiQuery] Screening expansion — 3 targeted queries (JSON schema):
• BRCA1/2 carriers NCCN guidelines mammography MRI surveillance ages
• risk-reducing salpingo-oophorectomy (RRSO) timing mastectomy
• hereditary breast and ovarian cancer surveillance protocols
4 queries × 15 results = 30 unique chunks
[PDF] Reranked 30 chunks against: NCCN cancer screening surveillance...
[CRAG] Grading 30 PDF chunks:
0.824 [CORRECT] BRCA PATHOGENIC/LIKELY PATHOGENIC VARIANT-POSITIVE MANAGEMENT
0.745 [CORRECT] NCCN Guidelines Version 3.2026...
0.699 [CORRECT] Bilateral Salpingo-Oophorectomy
... ← additional CORRECT chunks
0.009 [INCORRECT] → dropped
[CRAG] Kept 22 chunks | Dropped 8 INCORRECT chunks
Calling Ollama [ministral-3:14b] (3 DB + 10 PDF chunks)
Draft JSON generated (3200 chars)
Screening output (BRCA1-specific only):
• Annual breast MRI + mammography starting at age 25-29
• Risk-reducing salpingo-oophorectomy between ages 35-40
• ❌ NO colonoscopy (table row isolation prevented cross-gene slippage)
[Citation] Verified DB chunks: 3 | Verified PDF chunks: 10 → confidence: high
[LANGGRAPH FINISHED] Confidence: high
Completed:
-
PostgreSQL schema + ClinVar bulk ingestion with MD5 verification
-
pgvector setup + NCCN document indexing (Docling table-aware parser)
-
Query decomposition — keyword routing to typed SubQuery list
-
LangGraph 6-node DAG with parallel Send-based fan-out
-
Multi-query expansion with per-category prompt templates
-
BGE cross-encoder reranking (
BAAI/bge-reranker-large) -
CRAG evaluator with empirically tuned thresholds (0.05 / 0.01)
-
gnomAD v4 GraphQL client + BA1/PM2 deterministic annotation
-
ClinGen REST API client + gene symbol extraction
-
JSON-schema-constrained generation (ClinicalResponse Pydantic schema)
-
Citation manifest injection (prevents hallucinated references)
-
Anti-hallucination guardrails (Rules 5–6):
- Rule 5: Never invent variant biology (frameshift/missense/de novo/computational)
- Rule 6: Table row isolation — verify gene name in every NCCN table row
-
ClinGen validity field in ClinicalResponse schema
-
Screening/protocol mutual exclusion — prevents cross-category retrieval noise
-
ACMG removal — eliminated LLM-interpreted criteria to prevent clinical hallucinations; BA1/PM2 handled deterministically via gnomAD
-
Unit test suite — 4 test files covering routing, retrieval, generation, ClinGen
-
3-phase evaluation framework — Phase 1 (hallucination/citation), Phase 2 (Ragas faithfulness), Phase 3 (ablation)
-
28-case golden test set — multi-source, cross-gene traps, safe-failure, moderate-penetrance genes
-
Per-subquery reranking — chunks scored against focused topic text, not full mixed query
-
Reranker header stripping —
_score_text()strips[Header_N:]metadata prefix before BGE cross-encoder scoring, preventing bibliography chunks from scoring high due to keyword-rich NCCN section headers -
Full-fidelity Ragas contexts —
evaluate_nodenow stores untruncated chunk texts in trace;run_evaluation.pywrites them toraw_outputs_*.jsonso Ragas judges against the same evidence the LLM used -
Ragas bibliography filter —
_is_bibliography_chunk()inrun_ragas.pyremoves NCCN reference-section chunks (URL/numbered-citation heuristics) before building the Ragas Dataset -
Ragas context budget tuned — 600 chars/chunk × 4 chunks (was 350×5); covers full NCCN evidence chunks without hitting judge token limits
Remaining:
- gnomAD local cache (pre-fetch at index time, eliminate runtime external call)
- Clinical validation with domain experts
- Re-run Ragas after pipeline fixes for final dissertation-grade scores
This system is a clinical decision support tool only. It does not replace professional medical judgment. All outputs must be reviewed by a qualified clinician before use in patient care.
Built with ❤️ for safer, more transparent clinical AI