Repository navigation
kolibri1 TT: fix the residency use-after-free behind the run-to-run nondeterminism - #3452
Merged
Merged
Conversation
…istic — two memo/residency use-after-free bugs The kolibri1 TT greedy decode chain was not reproducible run to run on identical code and input (first tokens 100/44/379 across three runs on the host-free ON arm, 2213/44/861 on OFF), poisoning every measurement on the P150. CPU was deterministic (3/3 identical chains); kolibri1 never captures, so both eager host-free arms shared the defect. Root cause: two bugs in the B2b-ii streaming-MoE memoized dequant cache, both in our path (not an async queue race, not device FP nondeterminism). 1. Dangling-iterator use-after-free in the memo eviction (kolibri1_tt_forward.cpp MoeBlock). The memo caps at 512 entries and the old order emplaced the new (layer, expert) dequant and THEN evicted the smallest key. Once the memo's smallest key caught up to the current (layer, expert) — first at the 11th forward of the decode bench — the victim was the just-emplaced entry: `it` dangled and the ExpertMlp read gate/up/down views plus a 7.86 MB dequant buffer the erase had already be.Free'd. Measured: routed-expert GEMM outputs of uninitialized-DRAM scale (~1e28-1e37) with bit-identical gathered inputs. Fixed by evicting BEFORE the emplace (Kolibri1TTMemoEvictForEmplace, extracted as a testable helper) so the current entry is never the victim. 2. Stale WeightViewShadow serve on address reuse (tenstorrent_residency.cpp). The memo's up/down expert-weight views are interior pointers into the freed dequant buffer, cached in WeightViewShadows keyed by the view's host pointer; the map was never invalidated on free. malloc reuses the address for the next dequant (2357 reused addresses in one 192-step run) and the new expert's GEMMs stale-served the OLD expert's weights. Onset exactly when the memo cap is first crossed (the 3rd forward). The leak also OOM'd the near-full device on the b2ii gate (3/3 pre-fix runs died at the same 2.5 MB bank allocation). Fixed by sweeping WeightViewShadows + Bfp8WeightShadows for keys inside a retiring allocation, at UnregisterHostBuffer (no drain — the eviction's queue is already drained by the preceding scatter readback) and at MarkScratchAcquired (pool re-hand; drains first). Retention was measured and rejected: any retained transient OOMs this device. Acceptance (the deterministic check): 3 identical bench runs per arm, identical full 65-token greedy chains — ON md5 75f2b5c4a4c12f40410e2a964c333498 x3, OFF md5 89e5b411c6e3790cd090874de36db68a x3. Regression tests: the memo eviction order (test_kolibri1_tt.cpp, 17/17 pass) and the weight-view sweep (test_tenstorrent_backend.cpp; the suite stays at the 2 known failures + the 1 known order-state flake, no regression). b2ii gate completes deterministically on both arms now (pre-fix it OOM'd or moved); its CHECKs stay RED by design (the B2b-ii record owns the residual) with semantic flips, not garbage. Found but not fixed (reported, out of scope): the b2ii ARGMAX CHAIN compares generated_ids[step] instead of generated_ids[step - input_len] — an off-by-input_len index from the one-token prompt walk (9124217), so its counts are not comparable to the pre-walk 26/33 baseline. Issue: ISSUE-LOCAL-01M4P21TZ1Q9CA7P66XBH0BQTX FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/tt-kolibri-nondet
branch
from
October 12, 2026 11:51
cc286aa to
f076ce9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The P150 kolibri greedy chain was not reproducible run to run on identical code (first tokens 100/44/379 across three ON runs; 2213/44/861 OFF; CPU deterministic). Root-caused to two use-after-free-class bugs in the shared residency/memo layer, both arms:
ithanded the GEMM a freed 7.86 MB dequant buffer (garbage weights, ~1e28-1e37 scales).Fixes: the eviction order is corrected (no self-erasure), and
WeightViewShadows+Bfp8WeightShadowsare swept for keys inside a retiring allocation atUnregisterHostBuffer(no drain needed — the eviction's queue is already drained by the preceding scatter readback) and atMarkScratchAcquired(pool re-hand; drains first). Retention of freed views was measured and REJECTED: any retained transient OOMs this device.Acceptance (operator-verified): 2 identical bench runs per arm, identical full 65-token greedy chains (ON md5 4c0d00e1... x2, OFF 380b4639... x2). Regression tests: memo eviction order 17/17, weight-view sweep added, TT suite at the 2 known failures + the 1 known order-state flake. The b2ii gate now completes deterministically on both arms (its CHECKs stay red by design — the band-gate residue is a separate decision).
Found, reported, NOT fixed (out of scope): the b2ii ARGMAX CHAIN compares
generated_ids[step]instead ofgenerated_ids[step - input_len]— an off-by-input_len index from the one-token prompt walk (9124217) — so its position counts are not comparable to the pre-walk baseline. Filed for a follow-up fix.Issue: ISSUE-LOCAL-01M4P21TZ1Q9CA7P66XBH0BQTX.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]