Assessment: spaCy + scikit-learn query classification and adaptive RAG routing

[Copilot]

User proposed using spaCy and scikit-learn to classify query intent, then routing to full RAG vs. semantic-only retrieval based on type. No implementation was requested — this PR captures the technical assessment.

Findings

Current pipeline is one-size-fits-all
Every query — including greetings and simple definitions — runs the full retrieve → format → LLM chain. A classifier layer eliminates unnecessary retrieval and LLM calls for conversational and conceptual queries.

Recommended routing by query type

Type	Example	Pipeline
CONVERSATIONAL	"hello", "thanks"	LLM only — no retrieval
CONCEPTUAL	"what is CGPA?"	ChromaDB top-1 chunk — no LLM
PROCEDURAL	"how do I apply for leave?"	Full RAG
FACTUAL	"what is the MBA fee?"	Full RAG

Tool fit is good

• spacy en_core_web_sm (~12 MB) covers POS, dependency parsing, and NER — sufficient to distinguish "how many" (factual) from "how to" (procedural) and identify named-entity queries
• sklearn TF-IDF cosine similarity over per-category seed phrases is a lightweight fallback for ambiguous input
• Both sentence-transformers and ChromaDB are already present, so semantic-only mode adds zero new infrastructure

Key implementation notes (when ready)

• en_core_web_sm requires python -m spacy download en_core_web_sm post-install; needs a Dockerfile.api step
• Add spacy>=3.7.0 and scikit-learn>=1.4.0 to requirements.txt (no known CVEs on current releases)
• Initialise spaCy and the TF-IDF vectorizer as lazy singletons (mirror the existing threading.Lock() pattern in embedder.py) — classifier must not block the request thread
• Always fall back to full RAG when classifier confidence is low

Original prompt

check this project and tell me what kind of thing are used here like application stack and this session for future improvement of this project ok