Vector Space Information Retrieval Engine

A production-ready Vector Space Model (VSM) Information Retrieval engine built from scratch in Python using the Natural Language Toolkit (NLTK). This engine implements efficient text preprocessing, TF-IDF vectorization, an inverted index architecture, and an optimized top-K document retrieval strategy utilizing mathematical upper-bounding to prune the search space.

🚀 System Architecture & Features

This project mimics production-grade search engines by avoiding naive sequential scans over the entire document corpus. Instead, it utilizes a multi-stage data pipeline to achieve sub-millisecond query execution.

1. Document Preprocessing Pipeline

Every document and search query undergoes a strict tokenization and standardization pipeline:

• Case Folding: Converts all text to lowercase to ensure case-insensitive matching.
• Regex Tokenization: Extracts purely alphabetic words using NLTK's RegexpTokenizer.
• Stopword Removal: Filters out high-frequency, low-meaning words (e.g., the, is, at) using the NLTK English stopword matrix to reduce index noise.
• Morphological Stemming: Applies the Porter Stemmer algorithm to reduce words to their base linguistic roots.

2. TF-IDF Weighting Formulation

The mathematical importance of each token $t$ within a document $d$ is evaluated using the standard sub-linear log-frequency weighting scheme:

$$ w_{t,d} = (1 + \log_{10}\text{tf}_{t,d}) \times \log_{10}\left(\frac{N}{\text{df}_t}\right) $$

Each document vector is $L_2$ normalized to a unit length of 1 to ensure longer documents are not unfairly weighted during cosine similarity scoring.

3. Optimized Top-K Search & Pruning

Evaluating every document for a multi-word query is computationally expensive ($O(N)$). To bypass this, this engine implements an early-termination pruning strategy:

• For each query term, it inspects only the top-10 highest-weighted documents in its postings list.
• If a document is missing from the top-10 list of a query term, the engine computes a strict mathematical upper-bound score to safely discard non-viable candidate documents instantly without exhaustive scanning.

---

🛠️ Installation & Setup

1. Clone the repository:

   git clone (https://github.com/YOUR_USERNAME/vector-space-search-engine.git)
   cd vector-space-search-engine

2. Setup virtual environment:

python3 -m venv .venv
source .venv/bin/activate

3. Install dependencies:

pip install -r requirements.txt

4.Run

python3 main.py