Class 3 Homework - Data Extraction Toolkit

A multi-modal data extraction and preprocessing pipeline built with Gradio. It pulls text out of webpages, images, PDFs, and audio, then runs a cleaning pipeline (deduplication, language filtering, PII stripping, n-gram repetition removal) to produce a pretraining-ready dataset.

Features

Module	Description	Backend
Web Extraction	Extract main article text from a single URL or a batch of URLs	trafilatura
OCR	Extract text from images (.png/.jpg/.jpeg/.bmp/.tiff) and .pdf files	pytesseract + pdf2image
ASR	Transcribe audio files with auto language detection (GPU with CPU fallback)	faster-whisper
Data Preprocessing	HTML clean → language filter → MinHash dedup → PII strip → n-gram dedup	datasketch, langdetect, bs4

All extracted records are appended to JSONL files under output/ so they can be consumed by the preprocessing tab.

Project Structure

hw3/
├── main.py                  # Gradio UI entry point (4 tabs)
├── run_trafilatura.py       # Web text extraction
├── OCR.py                   # Image / PDF OCR
├── ASR.py                   # Audio transcription
├── preprocess_data.py       # Cleaning pipeline (MinHash, PII, etc.)
├── requirements.txt
├── EE_ARXIV_LINK.txt        # Sample arXiv URLs (Electrical Engineering topic)
├── test_data/
│   ├── PDF/                 # EE1.pdf ... EE9.pdf
│   ├── image/               # image.png
│   ├── audio/               # sample-0.mp3 ...
│   └── data/                # Fake_Pretraining_Texts.csv
├── output/                  # Generated JSONL files
│   ├── web_outputs.jsonl
│   ├── ocr_outputs.jsonl
│   └── asr_outputs.jsonl
└── pictures/                # UI screenshots

Installation

1. Create and activate a virtual environment:

   python -m venv venv
   .\venv\Scripts\Activate.ps1

2. Install Python dependencies:

   pip install -r requirements.txt

3. Install system dependencies:

   • Tesseract OCR — required by pytesseract. Download from the UB Mannheim builds and make sure tesseract.exe is on PATH.
   • Poppler — required by pdf2image for PDF rendering. Install via poppler-windows and add the bin/ folder to PATH.
   • ffmpeg — required by faster-whisper for audio decoding.

Usage

Launch the Gradio app:

python main.py

Open the local URL Gradio prints (typically http://127.0.0.1:7860).

Tabs

Web Extraction

Paste a single URL, or paste multiple URLs (one per line) into the Batch URLs tab. Extracted text is appended to output/web_outputs.jsonl.

OCR

Upload an image or PDF. Pages are rendered at 300 DPI and converted to grayscale before OCR. Results are appended to output/ocr_outputs.jsonl.

ASR

Upload an audio file or record directly through the microphone. The Whisper base model is used (GPU float16 if available, otherwise CPU int8). Results are appended to output/asr_outputs.jsonl.

Data Preprocessing

Upload one of the JSONL files produced above. The pipeline runs four steps:

1. Strip HTML tags and keep only English documents (langdetect).
2. Deduplicate near-duplicates with MinHash LSH (threshold 0.7, num_perm=128).
3. Replace PII (emails, credit-card-like numbers, phone numbers) with placeholders.
4. Collapse repetitive 3-grams that appear >= 3 times.

A sample of the first 200 cleaned documents is shown in the UI.

Output JSONL Schema

Each line in the output/*.jsonl files is a JSON object:

{
  "url": "https://arxiv.org/html/...",   // or "ocr": "<path>", or "asr": "<path>"
  "extracted_at": "2026-05-24T22:55:00",
  "content": "<extracted text>"
}

Running Modules Standalone

Each module can be run directly for quick testing (see the if __name__ == "__main__": block in each file):

python run_trafilatura.py
python OCR.py
python ASR.py
python preprocess_data.py

Notes

• Sample arXiv URLs for the EE topic are listed in EE_ARXIV_LINK.txt.
• The output/ directory must exist before the scripts append to it; create it manually if missing.
• ASR will auto-detect language but the printed log shows confidence — verify it on non-English audio.