[linson007]

Adds separate context, final-backbone and combined recall adaptation scopes while freezing the selector. Evaluation reports reached/covered/selected counts per position, so unvisited suffixes cannot inflate conditional recall. It also tests full QAT plus prefix/position objectives with recall training and exact resumed BF16 exports.

Merge scope and dependencies

Depends on #553; merge the training chain #499 → #537 → #551 → #552 → #553 → #554 in that order. All PRs target main per CONTRIBUTING; review this component against its immediate prerequisite. Parent changes remain visible against main until those parents merge. Merge commits preserve the tested ancestry; after squash/rebase merges, refresh dependent heads against main and rerun their checks.

Implements recall-training controls for #547 before adding capacity. Serving geometry is unchanged; larger convolution, fusion, rank/support or depth changes need trained evidence and native qualification. Ready-for-review status applies to these optional tools and their implementation tests. No checkpoint acceptance or speedup improvement is claimed, and no newly trained/experimental architecture becomes a serving default.

Validation at d066629ff370

• 76 local training/reference tests passed with no skips. Python 3.12.14, torch 2.13.0 and Transformers 5.15.1 on Apple M5 Pro/macOS 26.6.
• The pinned upstream numerical reference (07ebd93db9f472af339b644bb70221ad8428328a) ran. Actual MPS was required and CPU fallback disabled. Combined tests exercise context/full QAT, prefix/position objectives, recall scopes, finite CPU/MPS gradients, frozen selectors, recipe loading and exact resumed exports. The joint nine-PR merge rehearsal is conflict-free.
• Ruff lint/format, architecture boundaries and diff whitespace checks passed. CI checks the current head separately for CPU/reference training, Python 3.12–3.14, native production/CPU builds and sanitizers.

DFLASH_REFERENCE=/path/to/dflash/dflash/model.py \
DFLASH_REQUIRE_MPS=1 PYTORCH_ENABLE_MPS_FALLBACK=0 \
  python -m unittest discover -s dev/tests/training -t . -v

CPU/MPS fixture and reference checks above are implementation evidence. Additional real-checkpoint CUDA qualification is recorded below.

Real-checkpoint CUDA and native qualification

Recipes, model revisions, raw metrics and replayable native captures are published in #554. The CUDA source is the combined training stack at d066629ff370; this is additional integration evidence, not a claim that every isolated parent head ran on CUDA. One A100-SXM4-80GB ran the actual pinned BF16 Qwen3.8-27B teacher and released DFlash2 draft. Python 3.12.3, torch 2.13.0+cu130, CUDA 13.0, Transformers 5.15.1.

Baseline, QAT, prefix/position and combined recall each completed 20 steps. Recall preserved every frozen selector tensor exactly. The separate full-draft preflight passed, and both research-selector CUDA component tests passed. The new commit adds documentation and measured artifacts only; training source is unchanged from the tested d066629ff370.

All four 20-step recipes exported finite BF16 weights. Capture tensor payloads matched exactly across recipes; frozen export tensors were checked by payload hash. Validation mean matching proposals remained 3.875 → 3.875 in every run. All four exports passed native loading/capture/replay against a fixed low-bit target, with exact cloud export hashes verified. The released draft plus four exports produced 540 native lane-cycles. No consistent acceptance improvement or serving speedup was demonstrated. Tiny authored splits, one seed and correlated cycles limit effectiveness conclusions. The BF16 teacher and quantized native target remain distinct artifacts. No new checkpoint becomes a default; research/deployment issues remain open.

The pinned teacher template defaults to thinking enabled with xhigh reasoning effort; native probes explicitly disabled thinking. This mode mismatch, together with the BF16/low-bit target difference and short captures, prevents a matched-distribution effectiveness conclusion. Future capture controls should make thinking mode explicit and include it in cache/run provenance.

Final current-head checks

CI passed at 310e75d7f22e7a86f2141c5cf93e82aba62a9342: optional CPU/reference training, Python 3.12–3.14, production/CPU/native builds and sanitizers. GitHub reports MERGEABLE/CLEAN with no unresolved review threads. The latest nine-PR integration rehearsal remains conflict-free; only documentation/evidence differs from the previously tested implementation tree. The conditional release-hardware gate was not scheduled.

[linson007]

Completed real-checkpoint CUDA/native qualification

Tested training source: d066629ff370d94e645bbdd6d32f73a4c8194838. Published evidence: 310e75d7f22e7a86f2141c5cf93e82aba62a9342; the intervening commits add documentation/artifacts only. All 12 trainer-module hashes in the RunPod selection manifest match the published tested source. No unpushed training/runtime hotfix was required.

One A100-SXM4-80GB ran the pinned BF16 Qwen3.8-27B teacher and released DFlash2 draft (exact model revisions and software versions are in the report).

Recipe	Optimizer steps	Near-2K training peak	Validation matching proposals
Plain selector	20	10.6 GiB	3.875 → 3.875
Selector/context QAT	20	12.0 GiB	3.875 → 3.875
Prefix/position objective	20	10.6 GiB	3.875 → 3.875
Recall + QAT + prefix/position	20	16.2 GiB	3.875 → 3.875

All 80 steps, validation, checkpoint selection and finite BF16 exports passed. Captured tensor payloads matched exactly across the recipes. Export tensor hashes confirmed updates stayed within the requested scopes; every recall-run selector tensor remained exactly frozen. The recall recipe combines controls, so it is interaction coverage rather than an isolated recall ablation.

Teacher capture peaked at 51.8 GiB. A separate one-update full-draft preflight passed at 40.7 GiB; no 20-step full-draft run is claimed. Peaks are PyTorch allocated memory for the recorded near-2K probe, not total device requirements or larger-batch guarantees.

All four actual trained exports matched cloud export hashes, loaded in native Splash, and passed capture/replay on Apple M5 Pro. Including the released draft, the matrix contains 540 lane-cycles across four authored prompts. Non-draft input digests match across all five assemblies; target output common prefixes agree. The 20 sanitized published captures replay exactly. No consistent acceptance improvement was demonstrated.

Pair-context/nonlinear research-selector CUDA component checks also passed, but their artifacts remain research-only and are not the four native exports above.

This was an 8-train/4-validation/4-native-test authored smoke, one seed, with eight validation blocks and short continuations. The teacher used template-default thinking/xhigh; native probes disabled thinking and used a distinct 4-bit target. These differences and the tiny sample prevent a matched-distribution effectiveness conclusion. No serving throughput or sampled-decoding benchmark was run, and no newly trained weights become a serving default.

Current-head CI passed: optional CPU/reference training, Python 3.12–3.14, production/CPU/native builds and sanitizers. The conditional release-hardware gate was not scheduled. The nine-PR merge rehearsal is conflict-free; research/deployment gates remain open.

Full report and reproduction · summary.json · tensor-verification.json