Issues

• Modular pipeline: choose stages, quick scrape (no AI), tables, export any stage

  enhancementgap:pipeline
  ianktoo/data-forge#69 · Filter by author ianktoo opened on Sep 23, 2026

• Near-duplicates across pages: MinHash + LSH over the whole session

  enhancementgap:data
  ianktoo/data-forge#68 · Filter by author ianktoo opened on Sep 23, 2026

• Stream-parse sitemaps (iterparse, gzip) instead of loading them whole

  enhancementgap:pipeline
  ianktoo/data-forge#67 · Filter by author ianktoo opened on Sep 23, 2026

• Fetch a few pages at once in the fallback crawl and sitemap indexes

  enhancementgap:pipeline
  ianktoo/data-forge#66 · Filter by author ianktoo opened on Sep 23, 2026

• Input sources beyond websites: PDFs, documents, audio

  enhancementgap:data
  ianktoo/data-forge#48 · Filter by author ianktoo opened on Sep 23, 2026