Skip to content

[PR] kernel: preserve lifetime ref floor against dead-ref reuse (#5485) - #6904

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-11-16_rc-identity-msgid-ref-floor
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-11-16_rc-identity-msgid-ref-floor

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Root-cause fix: identity-msgid — kernel ref floor against dead-ref reuse

Fixes #5485

What this fixes

The identity-msgid root-cause batch (aggregation #6893) aggregates two issues:

  1. GitHub Historical ref floor and tombstones still permit lifetime reuse after #5360 #5485 — "Historical ref floor and tombstones still permit lifetime reuse after [PR] fix(kernel): post-restart compression no longer silently folds wrong messages — slots freed during ID reconciliation are | fix(kernel): 重启后压缩不再静默折叠错消息——编号对账时腾出的座位保留给原主(#5196) #5360": the kernel ref allocator can reissue a number whose previous owner was pruned (a byRef orphan), causing silent ref misattribution. This PR fixes it.
  2. Mirror [Bug][compress][externalSummary] Serial 25s/50s default budget makes long-payload chains fail 100%; non-fetch failures a | [Bug][compress][externalSummary] Serial 25s/50s default budget makes long-payload chains fail 100%; non-fetch failures are swallowe… #6885 (no GitHub counterpart) — cross-agent audit of the "session id = unique conversation identity" law. Verified accurate at the audit's base commit; needs no code change (recorded in the [Root Cause] identity-msgid (3 items) #6893 thread).

The fix — kernel core of #5233, rebased onto current master

The deadRefs tombstone infrastructure already landed via #5360. This adds the missing ref floor so a pruned number is never reissued:

  • highestIssuedIndex — a byRef orphan whose byRaw owner is gone still counts toward the floor
  • assignRefs — cursor clamped to max(requested, historicalHighWater + 1); a ledger stamp is accepted only when it lands above the floor and outside the tombstones
  • allocateFreeRef — skips tombstoned indices
  • both restampRefsToLedger settlement paths seed nextIndex from the unified floor

Cherry-picked from #5233 (qiangu2020) kernel core with authorship preserved. The host-layer hardening in #5233 (cold-load restamp retirement + async load fence) is a separate concern and is deliberately not included here — it stays in #5233 for its own review.

⚠️ Behavior change (old → new)

The ref floor clamps the assignRefs cursor above the highest issued ref, which changes how a caller-supplied nextIndex below the historical high-water mark is handled:

  • OLD: the cursor starts at nextIndex and claims the first free slot below it (a gap below the cursor is claimable).
  • NEW: the cursor is clamped up to historicalHighWater + 1; the gap below the mark is skipped, not claimed.
  • WHY: a low ledgerSeq / counter restart must not reopen the issued range — that is exactly the lifetime-reuse path Historical ref floor and tombstones still permit lifetime reuse after #5360 #5485 fixes (a number whose previous owner is gone from byRef would be reissued).
  • Consequence: tests/cache-theorem.test.ts [grounding:g1] re-pinned to the floor behavior (m00006 / nextIndex 7 instead of m00004 / nextIndex 5). This is the host-side consequence of the kernel floor; it does not change the kernel.

In practice the cursor is always highestUsedIndex + 1 (never below the mark), so this only affects the artificial low-cursor state the floor is designed to guard.

kernel-change: ref floor against dead-ref reuse (#5485)

The host side cannot solve this: the ref floor is an invariant of the kernel's ref allocator (assignRefs / restampRefsToLedger), which the host only calls. The defect is a design gap in the kernel — the allocator trusted highestUsedIndex alone, ignoring byRef orphans and tombstones — not a host-side workaround. Dedicated approval issue: #5485.

Verification

  • Kernel suite: 1651/1651 pass (incl. refs, seq-restamp, dead-refs, dead-ref-split)
  • Host seq-ledger / dead-ref / ledger-floor tests: 38/38 pass
  • cache-theorem: 5/5 pass (g1 re-pinned to the floor behavior)
  • typecheck + build clean

@ranxianglei ranxianglei changed the title kernel: preserve lifetime ref floor against dead-ref reuse (#5485) [PR] kernel: preserve lifetime ref floor against dead-ref reuse (#5485) Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-16_rc-identity-msgid-ref-floor (0fb805e)

Latest build — install this exact version

npm install -g billion-context@0.0.11017-pr.6904.2a29ddc

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
0fb805e billion-context@0.0.11017-pr.6904.2a29ddc build
98bd22a billion-context@0.0.10361-pr.6904.5057757 build
205595c billion-context@0.0.10344-pr.6904.ce91ee9 build
3fb54b6 billion-context@0.0.10340-pr.6904.0bb0627 build
d23b8dc billion-context@0.0.10339-pr.6904.3f401ea build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6904.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

lock-sync already fixed by an earlier wake — no further work needed.

Verified against live sources this run (not from memory / earlier comments):

  • The branch head has moved from the failing 3fb54b68 to 205595c3. Commit 205595c3 ("chore: sync kernel/package-lock.json version to 0.0.199", authored by the ci-check bot) already syncs kernel/package-lock.json version 0.0.198→0.0.199 — exactly the 2 lines the failing diff showed.
  • GitHub check-runs on the new head 205595c3: lock-sync → completed success. (The mirror cache still shows the old head's failure only because it was last fetched at 08:41, before the fix landed.)
  • I also re-ran the exact lock-sync logic locally on the new head (root + kernel npm install --package-lock-only + git diff --exit-code) — both clean, so the gate passes.
  • PR is mergeable (no conflict); blocked is the approval/pending-checks state, not a merge conflict.

Nothing to push — the fix is already committed and the check is green. Stopping without further work.

八问:

  1. 是什么问题: lock-sync CI 门禁红 — PR 把 kernel 版本从 0.0.198 升到 0.0.199(kernel/package.json),但 kernel/package-lock.json 里的 version 字段还停在 0.0.198,git diff --exit-code 非空。
  2. 什么原因导致的: kernel 版本升级(Kernel Boundary 强制 bump)后没有同步重新生成 kernel/package-lock.json,lockfile 记录的 version 与 package.json 不一致。
  3. 影响范围与严重程度: 仅影响该 PR 的 CI 门禁(lock-sync),不触碰任何运行时行为/代码逻辑;属 CI 卫生问题,非功能性缺陷。
  4. 怎么修复的: 把 kernel/package-lock.json 顶部 version 字段从 0.0.198 同步到 0.0.199(与 package.json 一致,即 npm install --package-lock-only 的产出)。
  5. 是否有回归问题: 无 — 仅 lockfile 2 行 version 字段,不涉及源码/依赖解析/行为。
  6. 是否破坏缓存: 不涉及(无缓存/重试/增量游标/去重键)。
  7. 是否改了设计方向: 不涉及。
  8. 是否引入了新的配置: 不涉及。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 同家族但机制不同。 本 PR 是 #5233 (qiangu2020) 的 kernel core cherry-pick,修 #5485 (ref floor against dead-ref reuse)。与在途 #5235 (#5196 slot-retirement / superseded quarantine) 同属 ref-identity 家族:症状对比——都是 ref 号被错误复用/归属;机制对比——#5196 是 slot 退役 + superseded 隔离(host 侧),本 PR 是 allocator floor + tombstone(kernel 侧)。不重复,各自独立。无其他 PR 针对 #5485。

Review:修复正确,但 PR 不完整——我已补一个 host grounding 测试 re-pin commit (98bd22ac3e)。

代码审查(kernel core 逐行读):

  • highestIssuedIndex 把 byRef orphan 计入 floor ✓
  • assignRefs cursor clamp 到 max(requested, historicalHighWater+1);ledger stamp 只在 ledgerSeq > floor && !byRef[text] && !reserved 时接受 ✓
  • allocateFreeRef 跳过 tombstone indices ✓
  • restampRefsToLedger 两条结算路径都从 unified floor 起 ✓
  • 正常流不变(caller 传 highestUsedIndex+1,无 orphan 时已高于 floor)

发现并修复的回归(PR 不完整): CI 两 lane 红(ubuntu-24-shard2、windows-22-shard1),同一测试 [grounding:g1] ref cursor mechanics pinned to assignRefs/allocateFreeRef(tests/cache-theorem.test.ts:289)。该 grounding 测试 pin 的是 pre-floor 机制("gap below cursor is the first free slot" → m00004);PR 加 floor 后同一 setup 现在分配 m00006(cursor clamp 到 highest issued+1 = 6)。PR 改了 kernel cursor 机制但漏更新这个 host grounding 测试 → CI 红。已本地验证(m00006 / nextIndex 7),补 re-pin commit 到 PR 分支,本地 5/5 grounding subtests 通过。

lock-sync: 作者已自修(3fb54b68f9 + 205595c384,三个 lockfile 同步到 0.0.199),本地验证过,无需再动。

CI: 刚推 grounding fix,CI re-run 中(30 checks);绿后我给最终判定。


结构化八问:

  1. 是什么问题:kernel ref allocator 在 byRef orphan(消息 prune 后 ref 号残留)或 tombstone(deadRefs)存在时,可能把已 prune 的 ref 号重新分给新消息 → 静默 ref 误归属(Historical ref floor and tombstones still permit lifetime reuse after #5360 #5485)。
  2. 什么原因导致的:allocator 只信 highestUsedIndex(byRaw 活绑定),忽略 byRef orphan 和 tombstone——floor 缺失。
  3. 影响范围与严重程度:特定条件(会话存在 byRef orphan 或 tombstone)下,新消息拿到已 prune 的 ref 号 → 静默误归属。p1(ref 错误改写类)。
  4. 怎么修复的:加 ref floor——highestIssuedIndex 计入 byRef orphan,cursor clamp 到 floor+1,ledger stamp 和 allocateFreeRef 跳过 floor 以下和 tombstone。
  5. 是否有回归问题:正常流不变。唯一变更:gap below highest issued 不再被 claim(以前填 gap,现在跳过)——floor 的 intended 行为(宁可不复用也不误归属);已补 grounding re-pin。
  6. 是否破坏缓存:不破坏。floor 只在分配时生效,不重排已有消息的 ref 号;新消息 ref 号可能更大(gap 跳过),一次性分配,稳定后不变。
  7. 是否改了设计方向:没改架构,只在 allocator 内部加 floor 不变量。
  8. 是否引入了新的配置:无。

判定:修复正确,PR 已补全(grounding re-pin)。等 CI 绿后建议合入。kernel change = human-only review,不 auto-merge。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已全绿。 head 98bd22ac3e(含 grounding re-pin)的 54 个 check run 全部完成:39 success、15 skipped、0 failure。之前红的两个 lane(ubuntu-24-shard2、windows-22-shard1)的 [grounding:g1] 测试已随 re-pin 转绿。

最终判定:可合入。

  • kernel core 修复正确(floor + tombstone,逐行审过);
  • PR 已补全(grounding re-pin 98bd22ac3e);
  • lock-sync 作者已自修;
  • CI 全绿。

kernel change = human-only review,不 auto-merge。请 owner 合入:#6904

qiangu2020 and others added 3 commits November 20, 2026 01:12
Cherry-picked from #5233 (qiangu2020) kernel core, rebased onto current
master. The deadRefs tombstone infrastructure already landed via #5360;
this adds the missing ref floor:
- highestIssuedIndex: byRef orphans whose byRaw owner is gone still count
- assignRefs: cursor clamped to max(requested, historicalHighWater+1)
- ledger stamp accepted only above the floor and outside tombstones
- allocateFreeRef skips tombstoned indices
- both restamp paths seed nextIndex from the unified floor

Prevents lifetime ref reuse after pruning (the #5485 defect).

Cherry-picked-from: 3e137c4 (qiangu2020/2026-10-30_ref-safety)
The #5485 ref floor clamps the assignRefs cursor above the highest
issued ref (a pruned number is never reissued), so a gap below the
highest issued is skipped rather than claimed. The grounding test
pinned the pre-floor behavior (gap below cursor is the first free
slot); re-pin it to the floor behavior (m00006 / nextIndex 7).

This is the host-side consequence of the kernel floor; it does not
change the kernel.
Rebase onto master (kernel 0.0.200) resolved the kernel version to 0.0.201;
the two lockfile-sync commits were skipped by the rebase as previously applied,
so regenerate both lockfiles to match: kernel/package-lock.json 0.0.200 -> 0.0.201,
root package-lock.json kernel ref 0.0.200 -> 0.0.201 (root stays 0.1.216).
@ranxianglei
ranxianglei force-pushed the 2026-11-16_rc-identity-msgid-ref-floor branch from 98bd22a to 0fb805e Compare November 19, 2026 17:15
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Merge conflict resolved and pushed.

The PR was mergeable: false (state dirty) — it had fallen 192 commits behind master. Master had moved the kernel version 0.0.198 → 0.0.200 (its b805e43 already consumed the 0.0.199 slot resolving the collision with #7252) while this PR's kernel change had bumped it to 0.0.199. That collided on kernel/package.json, both lockfiles, and kernel/src/compress.ts.

What I did this run:

  1. Rebased the branch onto origin/master (6453feee).
  2. Resolved the version conflict to kernel 0.0.201 (master's 0.0.200 + this PR's increment) — the correct landing point since 0.0.199/0.0.200 are already taken on master.
  3. kernel/src/compress.ts auto-merged cleanly: the PR's deadRefs: io.state.deadRefs line and master's Per-turn '[ACP status: …]' line keeps ordering the model to fold leftover small ranges right after a big fold — the opposite of what the compress receipt says #6928 removals (formatK, statusLine) sit in non-overlapping regions — both preserved.
  4. The rebase skipped the two lockfile-sync commits as "previously applied", so I regenerated both lockfiles to match (kernel 0.0.200 → 0.0.201; root kernel ref 0.0.200 → 0.0.201, root stays 0.1.216) and committed them as a separate chore: commit.
  5. Force-pushed (the authorized conflict-rebase force-push). Remote head is now 0fb805e4.

Verified locally before pushing:

  • lock-sync: clean (root + kernel npm install --package-lock-only + git diff --exit-code)
  • npm run typecheck: pass
  • npm run build: pass
  • targeted kernel tests (refs / seq-restamp / cache-theorem): 32/32 pass

The PR is now mergeable: true (state blocked = waiting on CI + review, no conflict). The new head's CI is running — the mirror cache still shows the previous head (98bd22ac, all green); I'll confirm the new head's checks once the poller picks them up.


八问

  1. 是什么问题:PR 与 base 分支冲突(merge state dirty),无法合并。
  2. 什么原因导致的:PR 分支落后 master 192 个提交;master 期间把 kernel 版本从 0.0.198 提到 0.0.200(b805e43 已占掉 0.0.199 解决与 [PR] compress.reasoning.tail: drop reasoning of any closed round in the protected window (#7144 follow-up) #7252 的冲突),而本 PR 的 kernel 改动 bump 到 0.0.199,两边在 kernel/package.json、两个 lockfile、compress.ts 上撞车。
  3. 影响范围与严重程度:仅该 PR 自身被卡住(blocked),不影响 master 或其他 PR;属体验瑕疵(PR 无法合并,需解冲突)。
  4. 怎么修复的:rebase 到 origin/master(6453feee);版本冲突按「master 0.0.200 + 本 PR 增量」解到 kernel 0.0.201;compress.ts 两处改动不重叠自动合并(保留 PR 的 deadRefs 修复 + master 的 Per-turn '[ACP status: …]' line keeps ordering the model to fold leftover small ranges right after a big fold — the opposite of what the compress receipt says #6928 删除);rebase 跳过的两个 lockfile-sync 提交手动重新生成对齐 0.0.201,单独成一个 chore: 提交。
  5. 是否有回归问题:无。compress.ts 的 union 同时保留 PR 修复与 master 删除;kernel 版本纪律满足(kernel/src 改动 + 版本 bump 0.0.200→0.0.201);lock-sync 干净;typecheck / build / 定向 kernel 测试(32/32)全过。
  6. 是否破坏缓存:不涉及(纯 rebase + 版本对齐,未触碰缓存/重试/增量逻辑)。
  7. 是否改了设计方向:不涉及(rebase 只对齐版本,不改设计/边界/数据流)。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值)。

判定:冲突已解决并推送,本地验证全绿;等新 head 的 CI 跑完确认即可合并(合并仍由人工执行)。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Historical ref floor and tombstones still permit lifetime reuse after #5360

2 participants