Corpus harvesting + ERID weak-label mining pipeline - #5
Conversation
The README's stated goal is reproducible corpus rebuilds; yt-dlp is a date-versioned, fast-moving dependency, so pin the resolved environment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7ab34ab03
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| try: | ||
| record = fetcher.fetch(url) | ||
| except Exception as exc: # one bad video must not abort the harvest | ||
| print(f"[skip] {url}: {exc}", file=sys.stderr) | ||
| continue |
There was a problem hiding this comment.
Because this catches every Exception from fetch, a systemic setup or code problem such as missing yt_dlp, bad fetcher options, or an extractor regression is reported as per-URL [skip] and main() still exits successfully with an empty or partial manifest. In the corpus-building workflow that can silently corrupt experiment inputs; catch only expected per-video download failures, or fail the run when the fetcher/environment is broken.
Useful? React with 👍 / 👎.
| name = "veritasad-research" | ||
| version = "0.1.0" | ||
| description = "Research tooling for VeritasAd: corpus harvesting and weak-label mining" | ||
| requires-python = ">=3.12" |
There was a problem hiding this comment.
The rest of the repo declares Python >=3.12,<3.13 (for example backend/pyproject.toml and the repository instructions), but this new package allows any future 3.x runtime. On machines with a newer interpreter installed, uv run can resolve and execute this tooling under an unsupported Python version, producing a lock/test environment that does not match the supported VeritasAd stack; add the same <3.13 upper bound here.
Useful? React with 👍 / 👎.
…ections 1-2) Brainstorm checkpoint for the next research milestone (M2): cached, versioned multimodal feature extraction (ASR/CLIP/OCR/MFCC) feeding model training and evaluation. Captures the four scoping decisions, the shared veritas-features core approach, the two-plan decomposition, and the agreed Sections 1-2. Sections 3-4 still being designed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Summary
Implements the corpus-harvesting + ERID weak-label mining pipeline (plan:
docs/superpowers/plans/2026-06-16-corpus-erid-mining-pipeline.md). A small, dependency-light Python packageresearch/, fully decoupled from the production backend:SCHEMA_VERSION, provenance (fetched_at).#реклама/#adhashtags, "на правах рекламы" phrase, sponsor phrases, promo codes.disclosed_adpositive set (keeps distant-supervision positives clean); weak-only signals stayunlabeledbut are flagged.MetadataFetcherProtocol +YtDlpFetcher; parsing is pure and fixture-tested, liveyt_dlpimport is lazy so unit tests never touch the network.existing_ids) /stats.docs/research/related-work.md(P0 literature-review tracking protocol),research/README.md, seed source list.uv-managed env with committeduv.lockfor reproducible rebuilds.Test Plan
cd research && uv syncuv run pytest -q→ 28 passeduv run python -m corpus.cli --help→ prints--sources/--manifestinterfaceyt_dlploaded (network isolation verified)research/data/sources.example.jsonlwith a real public video whose description contains anerid:marker, then run the CLI (manual — real URLs can't be fabricated)Known follow-ups (Minor)
research/README.mdlinks to../docs/research/2026-06-16-veritasad-research-roadmap.md, which does not yet exist in the repo (verbatim from the plan; create that doc or repoint the link).research/data/sources.example.jsonlcontains intentionalREPLACE_WITH_REAL_IDplaceholders for the live smoke run.🤖 Generated with Claude Code