WebSCI

An asynchronous web crawler, dynamic HTML renderer, and hybrid search engine powered by Playwright, SQLite FTS5, and ChromaDB vector embeddings.

WebSCI discovers web pages concurrently, renders client-side JavaScript applications using headless Playwright Chromium, extracts main body text using Trafilatura, maintains incremental synchronization using HTTP conditional headers (If-None-Match, If-Modified-Since), and serves hybrid keyword and semantic vector search via a CLI and a REST API.

---

Features

• CLI & Config Driven: Pass target seed URLs directly via CLI arguments (python main.py crawl https://example.com) or load environment settings from config.yaml.
• Dynamic JavaScript Rendering: Headless browser automation via Playwright Chromium with custom wait conditions and stabilization delays for single-page applications (SPAs).
• Asynchronous Frontier: Multi-domain URL traversal with domain politeness delays, per-domain concurrency semaphores, rate limiting, and depth limiters.
• Incremental Synchronization: Conditional HTTP fetching using If-None-Match (ETags) and If-Modified-Since headers to skip redundant downloads and embedding recalculations on 304 Not Modified responses.
• Raw Document Vault & Deduplication: SQLite-backed document store paired with raw HTML file retention and SHA-256 content deduplication.
• Hybrid Search Engine: Reciprocal Rank Fusion (RRF) combining sparse BM25 text rank (SQLite FTS5) and dense semantic vector similarity (all-MiniLM-L6-v2 via ChromaDB).
• FastAPI REST Service: Asynchronous background crawl job lifecycle management, telemetry metrics, document inspection, and search API endpoints.
• Database Maintenance Suite: CLI utilities for index integrity verification (verify), orphaned document repair (repair), and WAL database compaction (vacuum).

---

Architecture

┌────────────────┐     ┌────────────────┐     ┌────────────────┐
│   CLI / API    │ ──> │  URL Frontier  │ ──> │ Playwright DOM │
└────────────────┘     └────────────────┘     └────────────────┘
                                                       │
┌────────────────┐     ┌────────────────┐              ▼
│ Hybrid Search  │ <── │  FTS5 & Vector │ <── ┌────────────────┐
│  (RRF Engine)  │     │    Indexers    │     │  Trafilatura   │
└────────────────┘     └────────────────┘     │   Scraper &    │
        │                                     │  Page Storage  │
        ▼                                     └────────────────┘
┌────────────────┐
│  FastAPI REST  │
└────────────────┘

---

Installation

Prerequisites

• Python: 3.10 or higher
• Playwright Chromium: Headless browser binaries

Setup

# Clone the repository
git clone https://github.com/stackrift/WebSCI.git
cd WebSCI

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate
# On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install Playwright browser binaries
python -m playwright install chromium

Note

Installing Playwright Chromium binaries is required for rendering dynamic JavaScript pages.

---

Quick Start

1. Crawl Target URLs

Crawl any target website directly from the terminal without editing configuration files. WebSCI automatically derives the allowed domain host and applies parameters:

python main.py crawl https://fastapi.tiangolo.com/ --max-depth 2 --max-pages 50

Tip

You can also specify multiple target seed URLs at once:

python main.py crawl https://fastapi.tiangolo.com/ https://docs.python.org/3/

2. Build Search Indexes

Generates SQLite FTS5 inverted text indexes and ChromaDB vector embeddings:

python main.py index

3. Search

Execute a hybrid Reciprocal Rank Fusion search query:

python main.py search "asynchronous dependency injection" --k 5

---

CLI Reference

WebSCI provides a unified CLI in main.py.

Subcommand	Description
crawl	Run asynchronous web crawler with CLI seed URLs or config.yaml.
config	Preview runtime configuration with optional CLI overrides.
index	Build or update incremental search indexes.
reindex	Rebuild search indexes from scratch.
search	Execute hybrid search query.
compare-render	Compare raw HTTP fetch vs Playwright DOM output.
serve / api	Launch FastAPI web service on http://127.0.0.1:8000.
verify	Diagnose SQLite database and vector index consistency.
repair	Resynchronize orphaned index relations.
vacuum	Compact SQLite database and rebuild FTS structures.
frontier-demo	Run URL frontier queue demonstration.
crawl-demo	Run synthetic crawl simulation.
storage-demo	Run storage and deduplication demonstration.

crawl Options

python main.py crawl [urls...] [options]

• urls... (positional): One or more seed URLs to crawl.
• -u, --url <URL>: Target seed URL (can be repeated).
• -s, --seed <URL>: Seed URL alias (can be repeated).
• --seeds <URL1,URL2>: Comma-separated list of seed URLs.
• --max-depth <N>: Maximum depth allowed for link traversal.
• --max-pages <N>: Maximum total number of pages to process.
• -d, --allowed-domain <DOMAIN>: Allowed domain host filter (can be repeated).
• --allowed-domains <D1,D2>: Comma-separated list of allowed domains.
• --config <PATH>: Path to YAML configuration file (default: config.yaml).

Usage Examples

# Preview effective configuration with CLI overrides
python main.py config https://example.com/ --max-depth 3 --max-pages 100

# Rebuild search indexes from scratch
python main.py index --rebuild

# Search top 3 results
python main.py search "vector embeddings" --k 3

# Test Playwright rendering on a dynamic SPA page
python main.py compare-render https://quotes.toscrape.com/js/

# Run database integrity checks
python main.py verify

---

Configuration (config.yaml)

Default settings are declared in config.yaml and can be overridden at runtime via CLI arguments:

seeds:
  - https://fastapi.tiangolo.com/
  - https://docs.python.org/3/

allowed_domains:
  - fastapi.tiangolo.com
  - docs.python.org

max_depth: 3
max_pages: 500

concurrency:
  global: 5
  per_domain: 2

politeness:
  min_delay_seconds: 1.5
  user_agent: "WebCrawlerIndexer/0.1 (+contact: your-email@example.com)"

retries:
  max_retries: 2

fetch:
  robots_timeout_seconds: 10.0
  jitter_seconds: 0.5

playwright:
  wait_until: "domcontentloaded"
  navigation_timeout_ms: 30000
  stabilization_delay_ms: 1000

scraper:
  min_text_length: 200

indexing:
  embedding_batch_size: 32

embedding_model: "all-MiniLM-L6-v2"

---

REST API

Start the web service:

python main.py serve

The API listens on http://127.0.0.1:8000.

Endpoints

Endpoint	Method	Description
/health	GET	Health check endpoint.
/stats	GET	System stats, indexed page count, and active crawler config.
/metrics	GET	Runtime latency accumulators and job counters.
/search	GET	Execute hybrid search (?q=query&k=10).
/crawl	POST	Trigger asynchronous background crawl job ({"seeds": [...]}).
/crawl	GET	List all historical and active crawl jobs.
/crawl/{id}	GET	Get live execution status of a crawl job.
/crawl/{id}/results	GET	Categorized URL breakdown for a crawl job.
/documents	GET	Paginated listing of indexed documents (?page=1&page_size=20).
/documents/{id}	GET	Fetch metadata and raw text for a single document.

API Examples

Trigger Background Crawl

curl -X POST http://localhost:8000/crawl \
  -H "Content-Type: application/json" \
  -d '{"seeds": ["https://fastapi.tiangolo.com/"]}'

Query Search Endpoint

curl "http://localhost:8000/search?q=dependency+injection&k=3"

Important

To run WebSCI in an offline or air-gapped environment with locally cached models, set TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1.