Skip to content

feat: add safe exploratory training dataset builder - #298

Merged
shaggitza merged 1 commit into
mainfrom
feat/exploratory-training-dataset
Jul 29, 2026
Merged

feat: add safe exploratory training dataset builder#298
shaggitza merged 1 commit into
mainfrom
feat/exploratory-training-dataset

Conversation

@shaggitza

Copy link
Copy Markdown
Owner

Summary

Adds a lightweight exploratory PR-to-training-JSONL workflow, deliberately isolated from publication-grade benchmark custody.

  • hardens collect_prs.py as an offline-testable candidate-corpus collector with validated GitHub responses, strict UTF-8/JSON, controlled config errors, and no-clobber output;
  • adds build_training_dataset.py to join a corpus with validated completed legacy reviews or v2 compatibility projections while skipping unlabeled/explicitly non-evaluable rows instead of inventing negatives;
  • preserves legacy and v2 terminal status semantics, filters entrypoints by analyzer scope, and supports embedded, cached, or optionally fetched PR diffs;
  • adds a private POSIX directory-FD publisher for race-safe exclusive output/cache publication without following symlink path components;
  • validates full Git OIDs and confines deterministic cache names to the configured diff directory;
  • documents the separation between exploratory generation, frozen analyzer comparison, and publication-grade truth.

Safety and scope

  • No frozen corpus, review, adjudication, production, or v2 custody artifact is modified.
  • Output and fetched caches are exclusive-create only; there is no overwrite/--force mode.
  • Frozen artifacts, publication roots, input aliases, existing destinations, symlinked path components, and unsafe cache aliases are rejected.
  • JSON is strict: duplicate members, non-finite numbers, invalid UTF-8, and non-serializable output fail closed.
  • Completed labels require their expected fields and evidence; malformed statuses or duplicate identities fail instead of becoming empty targets.
  • The secure publication helper is private to the exploratory tools and intentionally relies on Linux/POSIX filesystem semantics.

Validation

  • 40 focused collector/builder tests passed, including path races, traversal, cache aliases, label/status validation, strict JSON/UTF-8, malformed GitHub responses, and first-fetch/cache repeatability.
  • All benchmark tests passed locally excluding only tests/benchmarks/test_pilot_typed_run_v3.py; four expected installed-runtime tests skipped.
  • Ruff format/check passed for all changed Python files.
  • Strict mypy passed for all changed Python files.
  • Two frozen-corpus smoke builds were byte-identical: 58 examples, 2 skipped labels, 58 missing diffs; SHA-256 d09e43404c71b8d769eeb94f3e693f2ec4426d3935839b477ec5e8ceafc06656.
  • Seven protected frozen artifacts remain byte-identical to HEAD.
  • Terminal independent acceptance review returned PASS / merge-ready.

Local limitation

The platform-specific native pilot broker suite remains unavailable locally because this host does not implement os.posix_spawn(..., setsid=True). GitHub Actions is the authoritative run for that existing suite.

@shaggitza
shaggitza merged commit 14fa1f7 into main Jul 29, 2026
6 checks passed
@shaggitza
shaggitza deleted the feat/exploratory-training-dataset branch July 29, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants