[linson007]

Adds bilinear, previous-hidden pair-context and candidate-conditioned nonlinear selectors for training-only comparisons at fixed rank/support and greedy walk. Zero-initialized additions reproduce the existing selector. Research artifact publication is atomic, and tensor contracts are checked before loading so failures leave weights unchanged.

Merge scope and dependencies

Depends on #552; merge the training chain #499 → #537 → #551 → #552 → #553 → #554 in that order. All PRs target main per CONTRIBUTING; review this component against its immediate prerequisite. Parent changes remain visible against main until those parents merge. Merge commits preserve the tested ancestry; after squash/rebase merges, refresh dependent heads against main and rerun their checks.

Implements architecture candidates for #546. Explicit capability-marked research artifacts cannot silently load as native serving checkpoints; native ports and real-model training are separate follow-ups. Ready-for-review status applies to these optional tools and their implementation tests. No checkpoint acceptance or speedup improvement is claimed, and no newly trained/experimental architecture becomes a serving default.

Validation at 0e496060667f

• 70 local training/reference tests passed with no skips. Python 3.12.14, torch 2.13.0 and Transformers 5.15.1 on Apple M5 Pro/macOS 26.6.
• The pinned upstream numerical reference (07ebd93db9f472af339b644bb70221ad8428328a) ran. Actual MPS was required and CPU fallback disabled. New failure-path tests verify interrupted research export cleanup/retry and rejection of malformed tensors before any weights are replaced.
• Ruff lint/format, architecture boundaries and diff whitespace checks passed. CI checks the current head separately for CPU/reference training, Python 3.12–3.14, native production/CPU builds and sanitizers.

DFLASH_REFERENCE=/path/to/dflash/dflash/model.py \
DFLASH_REQUIRE_MPS=1 PYTORCH_ENABLE_MPS_FALLBACK=0 \
  python -m unittest discover -s dev/tests/training -t . -v

CPU/MPS fixture and reference checks above are implementation evidence. Additional real-checkpoint CUDA qualification is recorded below.

Real-checkpoint CUDA and native qualification

Recipes, model revisions, raw metrics and replayable native captures are published in #554. The CUDA source is the combined training stack at d066629ff370; this is additional integration evidence, not a claim that every isolated parent head ran on CUDA. One A100-SXM4-80GB ran the actual pinned BF16 Qwen3.8-27B teacher and released DFlash2 draft. Python 3.12.3, torch 2.13.0+cu130, CUDA 13.0, Transformers 5.15.1.

Pair-context and nonlinear research selectors passed real-checkpoint CUDA component tests: exact zero-init baseline scores/paths, finite effective optimizer updates, frozen base and exact research-artifact reloads. Both remain native_compatible: false. This is not end-to-end training or a native implementation of either architecture.

All four 20-step recipes exported finite BF16 weights. Capture tensor payloads matched exactly across recipes; frozen export tensors were checked by payload hash. Validation mean matching proposals remained 3.875 → 3.875 in every run. All four exports passed native loading/capture/replay against a fixed low-bit target, with exact cloud export hashes verified. The released draft plus four exports produced 540 native lane-cycles. No consistent acceptance improvement or serving speedup was demonstrated. Tiny authored splits, one seed and correlated cycles limit effectiveness conclusions. The BF16 teacher and quantized native target remain distinct artifacts. No new checkpoint becomes a default; research/deployment issues remain open.

The pinned teacher template defaults to thinking enabled with xhigh reasoning effort; native probes explicitly disabled thinking. This mode mismatch, together with the BF16/low-bit target difference and short captures, prevents a matched-distribution effectiveness conclusion. Future capture controls should make thinking mode explicit and include it in cache/run provenance.

[linson007]

Real-checkpoint CUDA component results for the research selectors

The RunPod source was the combined training stack at d066629ff370d94e645bbdd6d32f73a4c8194838 in #554, which includes this PR and its prerequisites. This is integration evidence, not an isolated CUDA run of this PR's own head. One A100-SXM4-80GB ran the actual pinned BF16 Qwen/Qwen3.8-27B teacher and incoai/Qwen3.8-27B-DFlash2 draft (Python 3.12.3, torch 2.13.0+cu130, CUDA 13.0, Transformers 5.15.1).

Both pair_context and nonlinear passed tests using a real cached teacher continuation and released-draft hidden rows, including the anchor: hidden size 5120, rank 256, vocabulary 248320, top-16 candidates.

• Zero-initialized scores and greedy paths matched the baseline exactly.
• One effective optimizer update per variant produced finite gradients and changed trainable parameters.
• The baseline stayed frozen, and research-artifact save/load reproduced scores exactly.
• Both artifacts retain native_compatible: false.

These are component optimizer/artifact checks, not 20-step end-to-end training of either new architecture, native serving support, or evidence of better acceptance. The native matrix in the report uses compatible baseline/QAT/prefix/recall exports, not these research artifacts.

The exact PR head 0e496060667f91373c4eca81320ae9039d3dc226 separately has 70 passing local training/reference tests with actual MPS required and green scheduled CI.

Full report and reproduction · research-cuda-results.json