Skip to content

Add matched-prefix DFlash state restore probes and native evidence - #556

Open
linson007 wants to merge 4 commits into
incoai:mainfrom
linson007:experiment/dflash-restored-context
Open

linson007 wants to merge 4 commits into
incoai:mainfrom
linson007:experiment/dflash-restored-context

Conversation

@linson007

@linson007 linson007 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Adds a native fresh-versus-restored-prefix probe and a matched-context comparator for #548. The probe explicitly materializes a page-aligned prefix, snapshots the real composite state, moves the request to another state lane, restores it, rebuilds draft rings and resumes the same prompt. Fresh controls use identical materialization and prefill chunk boundaries.

Captured context hashes include the pending anchor. The comparator rejects mismatched model/prompt/precision/policy/budget settings, pairs only identical contexts and reports censored acceptance, proposal changes, verify-row choices, selector-confidence changes and optional BF16 final-hidden drift. Adaptive capture can freeze the prior/cost seed, avoiding online timing-learning confounds. Native scratch matches beyond retained/verified rows are censored.

Native evidence

The checked-in, hash-verified archive contains 12 captures / 48 cycles / 24 matched-context pairs on Apple M5 Pro (48 GiB, macOS 26.6): released dense and MoE DFlash2 drafts with their MLX 4-bit targets, 2112/8256-token source-review prompts, INT8/BF16 KV, and full/frozen-adaptive verify budgets.

  • Proposals and retained lengths were unchanged in all 24 matched pairs.
  • Final-hidden relative L2 maxima were 0.044–0.057 per case. Selector confidence moved even with unchanged proposals.
  • In one frozen-budget MoE cycle, fresh/restored chose six/five verify rows and both retained three tokens. This motivates calibration/cost measurement after restore; it is not evidence of lost acceptance or a speedup.
  • The treatment includes snapshot restore and lane movement. These data do not isolate context-window quantization from every numerical effect.

Validation

  • 26 capture/replay/comparison tests passed, including all archived file hashes and exact report reproduction, BF16 numerical drift, context mismatch rejection and adaptive-budget censoring.
  • B3/two-cycle capture passed with MTL_SHADER_VALIDATION=1, including hidden-row readback.
  • Nine native CLI error guards passed; the instrumented native benchmark and production library rebuilt successfully.
  • Production Runtime.cpp preprocesses byte-identically to upstream main (SHA256 11c4aac9285dae98a7f4db0f053072bfe83ac57beb99d77b3ed31c1130a77f38); extra accessors/reset hooks exist only in instrumented builds.
  • Ruff lint/format, architecture boundaries and diff whitespace checks passed.

Additional test limitation: combined MTL_DEBUG_LAYER=1 MTL_SHADER_VALIDATION=1 hit a prefill_gdn_scan_vh32 threadgroup-memory assertion (35072 > 32768) on this host. That extra check did not pass; it occurs in the unchanged target kernel, and the same assertion reproduced on the unchanged #550 baseline binary. Shader-validation-only capture passed. The failure is recorded in the validation artifact.

Dependencies and remaining gates

Ready-for-review follow-up to #550; merge #550 first. Based on ef0e95d4fa09f265f1cb071d43591ff32367b372; upstream main remains the base, so #550's parent capture changes are visible until it merges. Review only this follow-up here: experiment diff.

The research issue stays open. These are short source-review mechanics probes, not a broad long-context retrieval or task-quality benchmark. SSD/suspend restoration, larger contexts, direct ring reconstruction references, paired serving timings and any trained global-memory/window experiment remain unqualified. Capture I/O is unsuitable for throughput claims. No production architecture or checkpoint default is changed.

Trained-export native replay

Published CUDA/native evidence adds native loading and capture/replay of the released Qwen3.8-27B draft and four actual CUDA-trained BF16 exports: 540 lane-cycles on the same 4-bit target, INT8 KV, FP32 state, greedy decoding, all verification rows and frozen budget learning. All twenty sanitized published captures replay exactly. Non-draft input digests match across all five assemblies, and target output common prefixes agree in every comparison. No consistent acceptance improvement or throughput claim is supported by these four short prompts. This adds use of the existing diagnostics at b4006b12cdd7, without modifying its implementation.

The pinned teacher template defaults to thinking enabled with xhigh reasoning effort; native probes explicitly disabled thinking. This mode mismatch, together with the BF16/low-bit target difference and short captures, prevents a matched-distribution effectiveness conclusion. Future capture controls should make thinking mode explicit and include it in cache/run provenance.

@linson007

Copy link
Copy Markdown
Contributor Author

Additional current-head native capture/replay qualification

The actual native binary used the current PR head b4006b12cdd7e0eff0b759b2f5f1aa9d53e93610. It loaded the released Qwen3.8-27B DFlash2 draft and four actual CUDA-trained BF16 exports whose complete file hashes matched the cloud selection manifests.

  • 540 lane-cycles passed capture/replay: 108 per model, 32 each for code/math/prose and 12 before termination for JSON.
  • All 20 sanitized published captures replay to the exact archived reports.
  • All five assemblies used identical non-draft input digests. Target output common prefixes agreed in every trained-versus-released comparison.
  • Apple M5 Pro; fixed 4-bit target; INT8 KV; FP32 state; greedy decoding; thinking disabled; all verification rows; frozen budget learning.

This additional matrix used ordinary contexts (materialize_prefix=0, restore_prefix=0). It does not extend the restored-prefix/long-context qualification; the earlier matched restored-versus-fresh probes remain separate evidence. No serving throughput or consistent trained-model acceptance gain is claimed.

Full report and reproduction · native/results.json · native/comparison.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant