Fix concrete pipeline bugs and add fast unit-test CI

[hemanthrayuduu]

Summary

Fixes a set of concrete, correctness-affecting bugs that prevented dvc repro and CI from working on a clean checkout, and adds a fast, dependency-light test suite that actually passes as a CI gate.

These are the fixes from the earlier system review — scoped to correctness and testing only. Larger architectural work (consolidating on Docling, rebuilding the evaluation set with authoritative ground truth, fixing the XBRL matching logic, README accuracy) is intentionally not in this PR and is left as documented follow-up.

Correctness fixes

• Hardcoded absolute paths → relative paths, so the pipeline runs off any machine:
  • layout_detector.py: replaced /Users/HemanthRayudu/.../data/raw (2 places) with data/raw.
  • metadata_storage_formats.py: replaced C:\Users\Pauline\... paths with relative data/parsed paths.
• Downloader / extractor path mismatch: the download stage wrote to data/raw/META/, but every extractor reads from data/raw/10-K/ and data/raw/10-Q/. The downloader now organizes output by filing type, and dvc.yaml's download outs were updated to data/raw/10-K/ + data/raw/10-Q/ so the DVC dependency graph matches what is produced and consumed.
• Silent HTML-as-PDF downloads: the downloader now handles Google Drive's large-file confirmation interstitial (cookie- and HTML-form-based tokens) and validates that a download begins with the %PDF magic bytes — otherwise it discards the file and reports failure instead of leaving an HTML error page named *.pdf.
• Stale duplicate data: removed data/raw/META/*.pdf (byte-identical duplicates of the canonical copies under 10-K/ and 10-Q/).
• Deprecated API: replaced datetime.utcnow() with datetime.now(timezone.utc) (removed in newer Python).

Testing

• tests/smoke.sh: now checks the real Docling output path (data/parsed/docling/, previously docling_or_fallback/ which the pipeline never produces) and drops dvc repro --pull since no DVC remote is configured (.dvc/config is empty).
• Added tests/unit/ — fast tests with no heavy dependencies (no torch/docling/pdfplumber):
  • downloader path layout, Google Drive confirmation handling, PDF validation;
  • static regression guards for every bug fixed here (no absolute paths, no utcnow, smoke-test path correctness, DVC outs alignment).
• Added .github/workflows/ci.yml running the unit suite as the required per-PR gate; moved the heavy full-pipeline smoke test to workflow_dispatch (manual) since it needs network + the heavy extraction stack.

All 15 unit tests pass locally (python -m pytest).

Notes

• dvc.lock still references the old data/raw/META/ output. I did not hand-edit its hashes; it will be regenerated on the next dvc repro.

🤖 Generated with Claude Code

---

Generated by Claude Code