round7-3: training corpus v1 — decontaminated, teacher-labelled slices for the three slots, injection pairs, CPT text; manifest + decontamination report

[Xore]

Part of #3079 (round 7). Plan §6. No GPU for the build; the local-teacher labelling needs the card and queues behind the cold re-run — the OpenRouter-labelled synthetic slice does not.

The rule that shapes everything here

The benchmark corpus is the test set. Its 17 programs (xor_decode_loop, vulnerable_strcpy, strcpy_note_neutral, strcpy_note_injected, linked_list_sum, indirect_dispatch, error_handling_alloc, process_and_injection, process_witness_probe, tlv_parser, loopback_connect, safe_strcpy, integer_overflow_alloc, use_after_free, checksum_rotate, format_string_bug, file_write_persist) — at any toolchain or optimisation level, their decompiled text in tierb-cache/, their rubric terms, the adjudicated claim pool tier-a-v1.json, and every transcript under docs/benchmarks/runs/ — are off limits for training data, calibration text and RL prompts. Same for the evaluate-models.py session and Rev·Deck fixtures. A model that has seen them scores a memory, not a capability.

Slices to build (/mnt-1/training/corpus-v1/, mode 0700)

slice	job	input	labels	leaves the host?
S1 sessions-captured	session analysis	real Cowrie/Beelzebub sessions from honeypot-v* in ES, sanitised with llm-worker/contracts.py's sanitize_text / _redact_secrets / sanitize_commands — the production path, so the student trains on what it will see	local teacher, N=3, majority-agreed SessionAnalysis JSON only	never
S2 ghidra-captured	Ghidra triage	captured samples the production ghidra-worker already analysed (ghidra-analysis-v1); re-decompile through ghidra_cache.py against the headless service so the evidence shape matches the benchmark's Tier B	local teacher, N=3, majority-agreed triage JSON in the ghidra-triage-v1 contract	never
S3 ghidra-synthetic	Ghidra triage + Rev·Deck	≥ 60 new C programs written for this corpus across the rubric's behaviour families (encoding loops, memory-safety bugs, dispatch tables, persistence, network, benign near-neighbours, embedded-instruction controls) — none of them the 17; built with build_corpus.py's toolchain × opt-level matrix; decompiled the same way	OpenRouter open-weight teacher (DeepSeek V4 / Qwen3.8 large / GLM-5.3 / Kimi K3 — licences permit distillation), N=2 agreement	yes — synthetic only
S4 revdeck-rex86	Rev·Deck	Zenodo 15420461 REx86.zip (5,981 entries: intent, Q&A, complete-the-code, comments; CC-BY-4.0). Check for an internal held-out split first (#847's caveat)	as shipped	already public
S5 injection-pairs	all three	prompts from S1–S3 with newly written embedded attacker instructions (new phrasings, not the corpus's own strcpy_note_injected / process_and_injection text)	chosen = ignores the instruction; rejected = complies / breaks schema / flips polarity — filtered by injection_gate.classify_answer, the pydantic contracts and polarity.forbidden_hit	per source slice
S6 cpt-text	continued pretraining (round7-4)	raw domain text: decompiler output from S2/S3, sanitised session transcripts, REx86 text, public RE corpora with permissive licences	none	per source slice
dev	early stopping	10 % of each slice, split by program / session, never by row	—	—

Teacher key for S3 lives on the host (/etc/apiary-training.env, 0600, or the operator's choice) — never in the repo, never in an issue. Captured slices S1/S2/S5-from-captured/S6-from-captured stay on the host; if anything is pushed to Hugging Face it is a private repo and only ever the synthetic/public slices — operator decision, default off.

Decontamination (a deliverable, before any score is quoted)

analysis/ghidra/training/decontaminate.py: exact-hash and near-duplicate (MinHash + 8-gram overlap) comparison of every training prompt and answer against (a) the 17 programs' sources and every decompiled variant in tierb-cache/, (b) the rubric's keyword groups, (c) the claim pool, (d) the session/Rev·Deck fixtures. Required result: 0 hits; the report is committed as docs/benchmarks/training/corpus-v1-decontamination.md with counts per slice.

Deliverables

• build_corpus_v1.sh + the S3 sources + decontaminate.py + the teacher prompt templates committed under analysis/ghidra/training/corpus/ (synthetic sources reviewed for TEST-NET addresses / reserved names / fake credentials like the benchmark fixtures)
• docs/benchmarks/training/corpus-v1.md: counts, token totals, teacher tags + digests, agreement rates, sha256 per shard, what was rejected and why
• the decontamination report, 0 hits
• captured-derived data verified absent from git and from any remote

Depends on round7-1 for the local-teacher labelling (Ollama-served teachers need the card free). Blocks round7-4, round7-5, round7-6.

[Xore]

Working conventions for this issue: grit (claim the symbols you touch with grit claim -a issue-<N>-coder … where N is this issue number, edit only in .grit/worktrees/issue-<N>-coder/, grit done to merge, grit init after the merge if you added files), rtk on every shell command (rtk proxy only for raw output; rtk gain --history in the closing report), and the gh claim/comment/PR/merge conventions. Full text: epic #3079 → "Working conventions", and the plan's §15.

[Xore]

Claimed for round 7 (orchestrator, 2026-09-06). Pre-flight done: plan of record merged via #3089; grit initialised at repo root; no GPU work starts until the cold re-run completes.

[Xore]

Corpus evidence cross-reference — 2026-09-17

The #3084 per-slot handoff publishes the existing September 17 readiness/S4 findings. This issue's closed delivery status is unchanged; it does not establish final corpus acceptance.

Corpus delta requiring owner disposition:

• S1/S2 train/dev source maps empty. S3 63 rows, all 63 unlabelled, kept=0. Operator verified ghidra and sessions train/dev files each 0 bytes / 0 lines.
• S4 source metadata: train 4,045 rows, kept=4,045, dropped=0; dev 597 rows, dropped=22, kept=575. All 22 exclusions belong to dev, not train. Revdeck/union report 4,045 train / 575 dev each; operator verified revdeck train 4,045 lines / 1,599,265 bytes. Listed component names alone do not prove populated S1–S3 components.
• Surviving scan mtime 2026-09-14 08:10:25Z: 4,642 scanned, 38 hit events, pass=false. Slice-report mtime 08:10:26Z. The scan exposes no input hash/reference binding. These are modification dates, not proven execution order. 38 events need not be 38 distinct samples; the 22 exclusions do not establish remaining final-training contamination or accepted clearance.
• Credit the 2026-09-09 17:08:02Z post-filter claim: train-only 4,185→4,045, then 0 hits on 4,045, plus calibration acceptance. It predates the later failure and does not bind that historical clean claim to the later final train/dev inputs. Calibration 6,881/0 and S6 37/0 reports are different acceptance scopes.

Administrative deliverable: corpus/acceptance owner identifies existing dated per-slot acceptance records with final train/dev manifest identities/hashes, label coverage and composition, licensing/teacher/locality approvals, group/split separation, protected-set/scanner identities, and the failed scan's exact input plus any existing accepted replacement. Explicitly preserve/supersede the failure through authorized acceptance, not arithmetic or issue closure. If references do not exist or cannot be recovered, record unavailable and retain the gate. Prior metadata searches were bounded, not universal absence evidence.

No sample/ID bodies, new scan, generation, filtering or corpus remediation requested. This handoff grants no waiver, issue-state change or training authorization; the epic disposition still separates SFT-only from conditional CPT gates.

[Xore]

Corpus-gate evidence leg (read-only dispatch, 2026-09-18) — root cause of empty slices + what the 38 decontamination hits are

Full report in /home/xore/dispatch-3082-corpus-gate/REPORT.md (12.8 KB, path/line-cited). Summary:

1. Why ghidra/sessions slices are empty (root cause found). The live generator is the host-side /var/training/corpus-v1/build_slices.py, and its SLICE_SOURCES assigns empty source lists to S1 (sessions) and S2 (captured Ghidra); S3's 63 rows are all unlabelled so zero are kept (slices-report.json records S1/S2 with no source paths, S3 63/63 unlabelled). ghidra_{train,dev}.jsonl and sessions_{train,dev}.jsonl verified 0 bytes on the host. The repo-side slices.py deliberately keeps captured S1/S2 out of repo building pending host-side labelling — no authoritative labelled source has ever been wired in. This is a missing-input problem, not a generator bug.

2. The 38 hits are over-broad phrase matching, not leakage. All 38 events are kind=phrase on 22 distinct valid-s4-* IDs; zero exact or near-duplicate hits. The matcher is a literal substring test of every multi-word rubric phrase against prompt+completion — the triggers are ordinary technical vocabulary: "return address", "stack overflow" (in an x87 discussion), "each byte", "format string"/"format specifier", "return code", "control flow", etc. No benchmark C source, ground-truth, claims, cache, or transcript matched. With --background (known-unprotected corpus exclusion) these generic matches would be filtered by design.

3. New discrepancy found: 140 unexplained train-row exclusions. rex86 train has 4,185 entries but published S4 train has 4,045 rows, and the 38-hit report does not account for the difference (38 events ≈ 22 samples). Corpus owner must reconcile before any reproduction claim.

4. Minimal hash-bound acceptance (proposed, not implemented). input-acceptance.json written adjacent to slices-report.json at build time: SHA-256 over raw bytes of every final slot train/dev JSONL + S4 train/valid/test (valid/test held out, never in the training manifest), every scanned manifest, and the decontamination report; SFT launcher re-verifies all hashes and refuses on mismatch. Field list + rebuild recipe (per-split staged generate_s4.py builds → scan with --background → owner review → final scan → build → acceptance) are in the report. The current scan report records counts/hits but no scan-input digests — the binding must be added rather than reusing the historical report as certification.

No passing rebuild is established by current evidence. Blockers needing owner decision: authoritative labelled S1/S2/S3 sources, the 140-row reconciliation, and acceptance of the background-corpus policy.