Skip to content

[PR] fix(storage): persist the turn-final reply at turn end — it only landed on the next client re-send (#6947) - #6953

Merged
ranxianglei merged 13 commits into
masterfrom
2026-11-16_handoff-reply-turn-writeback
Nov 17, 2026
Merged

ranxianglei merged 13 commits into
masterfrom
2026-11-16_handoff-reply-turn-writeback

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Symptom (live)

In pi sessions the model's FINAL reply of a cleanly-finished turn (stopReason=stop) was missing from index.db — and therefore from the /__bili web UI, exports, and restart restore — until the NEXT request re-sent the conversation. While the session idles, the row never arrives.

Live evidence (both verified against ~/.local/share/billion-context/sessions/index.db + the session jsonls):

  • 01a1dd98-8a90-7d6a-bed5-abb12d3978c4: turn ended ~19:44:50 (+08) with the reply "都是正常的环境类,零候选回归。…" (jsonl: stopReason=stop, output 1157 tokens); DB max seq 2441 = the tool result preceding it; zero DB rows for the reply text; no new rows for 55 minutes.
  • 01a1de38-fedb-70ca-b573-7a5736fd3543: same signature — query at 20:44 found nothing; after the 20:59 user message the row landed (seq 1337).
  • Non-inline check: 01a1dd94 (bili pi -r) shows the same gap for its 13:44 final reply → the write path is shared, not pi-native specific.

Root cause

session.lastMessages (the only thing the store ingests) is replaced per request by snapshotMessages() (src/session.ts:1743) from the CLIENT's raw request array. The turn's own reply exists only in handoffReply → session.metadata.handoffResponse, whose sole consumer is the bili export CLI. So the reply row appeared exactly when the client re-sent it — the next request. markDirty() at turn end schedules a save, but the save has nothing new to write.

Changes

  1. src/server/handoff-response.ts — save(), after all existing guards pass, appends the captured messages back into the captured session.lastMessages array with codec-derivable ids (deriveMessageId + per-conversation cluster numbering). The debounced turn-end save then persists them immediately.
    • Folded snapshots (lastMessagesFolded) are never grown (folded-view contract); metadata capture is unchanged, so bili export output is byte-identical.
    • All stale guards (request advance / snapshot swap / discard / oversize) suppress the write-back exactly as they suppress the metadata write.
  2. src/storage/store.ts — in the full-ingest remap, when a plan row JOINS an existing stored row whose ref is still NULL (its ref was minted only once the client re-sent the tagged text), backfill it. NULL only: a stored non-null ref is the index-time projection and never rewrites; the "BLOCKED" byRaw-only token never lands in messages.ref.

Why the ids join (regression reasoning, §7.3)

The store join key is ${msg_id}\u0000${identity} (store.ts, ingest.ts). identity derives from role|contentType|toolName|canonicalText — the write-back uses the SAME content the production codec would derive on the re-send, and deriveMessageId with cluster numbering reproduces the codec's id for the Nth identical message in the same conversation. Test a client re-send through the production codec derives the SAME ids pins exactly this against the real openaiToCore pass. If the join hits: UNIQUE keeps one row, seq is stable (D4-1). If it misses (future client lane pins its own ids, #4491, or a protocol switch re-renders the text): worst case is one bounded extra row, the refs ledger still maps by raw id (store.ts:1467-1469), and the full-ingest tail sweep RETIRES (never deletes) the superseded identity — self-healing on the next save, never wrong rows (#4182 class).

Model-visible effect (intentional, noted per §7.3): after a restart, the restored conversation now INCLUDES the turn-final reply where it previously dropped it — the model no longer loses its own last answer across restarts. Nudge/footer range listings now see the reply text as compressible one request earlier than before.

Verification

(ai: qwen3.8-27b)

ci-check and others added 3 commits November 16, 2026 20:42
…ed on the next client re-send (#6947)

The handoff capture (handoffReply) wrote its messages to
session.metadata.handoffResponse — consumed solely by the 'bili export'
CLI. The store rows existed only after the NEXT request re-sent the
conversation, so an idle session's last reply was missing from
index.db and the /__bili web UI kept showing the previous reply
(live sessions 01a1dd98 and 01a1de38: reply in pi jsonl with
stopReason=stop, zero DB rows, none arriving while the session idles).

- handoffReply.save(): append the captured messages back into
  session.lastMessages with codec-derivable ids (deriveMessageId +
  cluster numbering), so the debounced turn-end save persists them
  immediately. Folded snapshots are untouched; all stale guards
  (request advance / snapshot swap / discard / oversize) apply to the
  write-back as well.
- store.ts: a full re-ingest that joins a stored row whose ref is
  still NULL backfills the minted ref (NULL only — the stored value is
  the index-time projection and never rewrites; the 'BLOCKED'
  byRaw-only token stays out of messages.ref).
- tests/issue6947-turn-final-writeback.test.ts: id stability vs the
  production codec re-send (the store join key), cluster numbering,
  tool-call fields, folded/stale no-ops, and store-level
  land-once + backfill guards.
@ranxianglei ranxianglei changed the title fix(storage): persist the turn-final reply at turn end — it only landed on the next client re-send (#6947) [PR] fix(storage): persist the turn-final reply at turn end — it only landed on the next client re-send (#6947) Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-16_handoff-reply-turn-writeback (80daffc)

Latest build — install this exact version

npm install -g billion-context@0.0.10545-pr.6953.f3a8c0b

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
80daffc billion-context@0.0.10545-pr.6953.f3a8c0b build
31d76f0 billion-context@0.0.10533-pr.6953.f491f81 build
c0fb8d1 billion-context@0.0.10474-pr.6953.b4e8b5a build
b93aa5e billion-context@0.0.10457-pr.6953.9a8b47c build
4453de8 billion-context@0.0.10447-pr.6953.ad69c34 build
b5de5cd billion-context@0.0.10437-pr.6953.a8e7a7d build
c0242dc billion-context@0.0.10435-pr.6953.6d0671d build
07bf7e8 billion-context@0.0.10433-pr.6953.a6182e8 build
1ae8926 billion-context@0.0.10431-pr.6953.f151b07 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6953.tgz
npm install -g package

Install instructions are refreshed on each push.

…oin exactly (#6947)

The original write-back derived ids with legacy deriveMessageId
(toolCallId slot filled, raw args text). The production codecs feed
deriveCanonicalMessageId (#3642): toolCallId pinned empty, args via
canonicalToolArgs. Plain text/reasoning seeds coincide, but every
tool-call write-back row minted an id the re-send pass would never
derive — a (msg_id, identity) join miss per tool call: refused delta
+ O(history) full re-ingest + a retired-row tombstone, every turn,
on exactly the long sessions this project exists for.

- new kernel helper baseIdOfCore(BiliMessage): re-derives the canonical
  base id from core fields (null only for image-part cores, whose #3751
  payload fingerprint lives in sidecars the capture never sets);
- the handoff write-back pre-seeds a ClusterCounter over the stored
  tail with it, so appended ids number like the next full codec pass;
- tests: the tool-call write-back id is pinned against the canonical
  derivation, plus a re-send-join test through the real openaiToCore
  (text + tool-call + tool-result) that fails under the legacy seed.

Verified: issue6947 suite 9/9; handoff-response/handoff-capture-work/
handoff-client-abort/web-handoff-roles/e2e-export-handoff +
storage-ingest-watermark/issue5744-ingest-diff/issue5794-ingest-cap/
order-unified-store/issue4665-archive-restore-identity = 74 pass /
0 fail / 1 skip; kernel suite 1640/1640; 20-suite storage sweep
139 pass / 8 fail — the 8 reproduce identically on clean master
(pre-existing content-store envelope failures, unrelated);
tsc --noEmit and npm run build clean.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status report for PR #6953 — the previous session was interrupted right after opening the PR (no final report landed in-thread). This run verified the branch and unblocked CI.

Verified locally (targeted runs):

CI: first run was green except architecture-docs — an inherited master red: reference/architecture.md missing: src/responses-system-sections.ts (file added by #6944 without the module-map update; master 301e1a5459 fails the same check locally). Per the CI coordination policy I merged ci-fixes (sentinel's fix 8ed56be997) into the PR branch — new head 07bf7e87b5, the docs gate is green locally, CI is re-running. One-line note posted in #4599.

Labels on the PR (judged on its own diff): bug, p2, scope:storage, root-cause.

八问:

  1. 是什么问题:pi 会话一轮干净结束时,模型的最后一条回复不会写入 index.db(web UI / 导出 / 重启恢复都看不到),要等下一次客户端重发整个会话才落库;会话空闲时该行永远不来。
  2. 什么原因导致的:session.lastMessages(store 唯一摄取源)每个请求都被 snapshotMessages() 从客户端原始请求数组整体替换;本轮回复只存在于 handoffResponse 元数据(唯一消费者是 bili export),所以轮末的 markDirty() 保存无新内容可写。
  3. 影响范围与严重程度:所有 pi 会话(写路径共享,bili pi -r 非内联车道同样复现);每轮最后一条回复在下次请求前从 web UI / 导出 / 重启恢复中缺失——数据可见性缺陷,非数据丢失(下次请求即补齐)。
  4. 怎么修复的:在决策点修——handoffReply.save() 把捕获的回复以「生产 codec 重发时必然推导出的同一 id」(5 槽 seed + 簇编号)追加回快照,轮末防抖保存立即落库;重发时 store 按 msg_id+identity join 命中同一行,UNIQUE 保一行;store 侧对 join 命中且 ref 仍为 NULL 的行回填铸出的 ref(单向、幂等、拒 BLOCKED)。
  5. 是否有回归问题:审过的既有行为——折叠快照不增长(folded-view 契约)、四条 stale 守卫(request 前进/快照换绑/discard/oversize)同样抑制写回、bili export 输出字节不变(元数据路径未动)、ref 回填只 NULL→铸值且语句内二次判 NULL 保幂等、seq 稳定性(D4-1)有测试钉住;store.test.ts 的 8 个失败为 master 既有(已对照验证)。
  6. 是否破坏缓存:不涉及缓存/重试/增量游标/去重键——store join 键与 seq 规则未变,仅新增 NULL ref 回填。
  7. 是否改了设计方向:无架构转向,写路径不变,只是轮末保存有了可写内容。有意且已在 PR 正文披露的模型可见变化:重启恢复的会话现在包含轮末回复(以前会丢);nudge 范围列表提前一个请求看到该回复文本。
  8. 是否引入了新的配置:无。

判定:代码已核对、继承性 CI 红已通过 ci-fixes 解除,等重跑全绿即可合并。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

评审结论:方向对、主 case(纯文本回复)修得干净,但有一处实际缺陷——工具调用回复的 id 推导与生产 codec 不一致,长会话里每个工具调用回合都会多付一次全量重 ingest。已直接修进本分支:commit c0242dc7,本地全绿,等 CI。

找到的问题(已修)

原实现的 baseIdentityId 用 legacy deriveMessageId(seed 里 toolCallId 槽填值 + 原始 args 文本),而生产 codec(openai/anthropic/google/responses 四条)喂的是 deriveCanonicalMessageId(#3642:toolCallId 槽钉死为空、args 走 canonicalToolArgs 归一化)——wire 上的 toolCallId 是传输管道,不是内容身份。

后果按类型拆开:

  • 纯文本/思考回复:两种 seed 字节相同 → join 精确命中 ✓(#6947 观察到的 case 不受影响)
  • 工具调用回复(pi 会话里每回合都有):write-back 行的 id 和客户端重发时 codec 推导的 id 不同 → (msg_id, identity) join miss → delta 快速路径被拒 → 每回合一次 O(历史长度) 的全量重 ingest + 一条 retired 墓碑行(retired 行的文本 CAS ref 永不释放)。会话越长每回合越贵——恰好砸在这个项目最核心的长会话场景上。

修法(最小改动,不碰 store.ts 与现有 guard):

  1. kernel 新增 baseIdOfCore(BiliMessage)(kernel/src/wire/identity.ts):从 core 字段重推 canonical base id,唯一返回 null 的情况是 image-part core(其 #3751 payload 指纹在 sidecar 里,capture 从不设置 imagePart,实际不可达);
  2. handoff-response.ts 的 write-back 改为用它在已存 tail 上预置 ClusterCounter,追加行拿到的 _N 序号与下一次完整 codec pass 完全一致;
  3. 测试补强:工具调用 write-back id 断言改成对 canonical 推导;新增"工具调用回复经真实 openaiToCore 重发后 id 逐字节相同"的 join 测试(text + tool-call + tool-result 三连)——这个测试在原实现下是红的,钉死了 exact-join 属性。

核实过没问题的部分

  • 持久化链路:debounced turn-end save 确实把 tail 写进统一索引(src/persist.ts guardedBuild 里 shadow.record(record, { lastMessages: tail })),回合结束即入库,不等下次请求 ✓
  • folded 快照 / stale guard(request 前进、snapshot 换数组、discard、oversize)对 write-back 的压制与 metadata 捕获完全对齐;bili export 输出不受影响(metadata 仍用 capture 自己的 reply_* id)✓
  • store.ts 的 NULL-ref 回填:方向正确(写回行入库时 ref 为 NULL,重发 ingest join 命中才补),guard 齐(NULL-only 不覆盖已有值、拒 "BLOCKED" token、语句内 ref IS NULL 复查保幂等)✓
  • incremental-lane 会话无重复风险:lane 自己就是 canonical-id 机制(incremental-lane.ts 注释 "stamp the canonical ids so prepare pins exactly what the lane owns"),与本修复同构,不存在 write-back 行与 lane 行双份的问题 ✓

验证数据(本地,commit c0242dc)

  • tests/issue6947-turn-final-writeback.test.ts 9/9(含新工具调用 join 测试)
  • 相邻 10 套件(handoff-response/handoff-capture-work/handoff-client-abort/web-handoff-roles/e2e-export-handoff + storage-ingest-watermark/issue5744/issue5794/order-unified-store/issue4665)= 74 pass / 0 fail / 1 skip
  • kernel 全套 1640/1640
  • 20 个存储套件 sweep:139 pass / 8 fail —— 这 8 个(store.test.ts 的 content-store envelope 家族 #2674/#2298)在去掉本 PR 全部改动的 clean master 上原样复现,预存环境问题,与本 PR 无关
  • tsc --noEmit、npm run build 干净

合并留给人。CI 我刚推完 fix commit 正在跑。

…guard #2092)

kernel/ is human-gated: a PR touching it must declare 'kernel-change' in
its body. The helper is host plumbing — it only pre-seeds the write-back
ClusterCounter and needs two helpers the kernel ALREADY exports
(deriveCanonicalMessageId, canonicalToolArgs). Moved baseIdOfCore into
src/server/handoff-response.ts as a module-local function; the PR diff no
longer touches kernel/.

Verified: tsc --noEmit clean; issue6947 suite 9/9; handoff-response/
handoff/handoff-capture-work/handoff-client-abort/web-handoff-roles/
e2e-export-handoff = 44/44; check:unused-exports clean (12 waived).
@ranxianglei

Copy link
Copy Markdown
Owner Author

补充(接上一条评审):CI 的 kernel-guard(#2092)拦了 c0242dc——它规定动 kernel/ 目录的 PR 必须在正文声明 kernel-change: <reason>(kernel 改动是人工门)。

处理方式:与其走声明流程,把 baseIdOfCore 从 kernel/src/wire/identity.ts 挪回 host 侧(src/server/handoff-response.ts 模块内函数)——它只是给 write-back 预置计数器,依赖的 deriveCanonicalMessageId / canonicalToolArgs kernel 本来就导出,挪过去语义完全一样,PR diff 不再碰 kernel/ 一个字节,守卫自然过。

  • 新 commit:b5de5cdd(在上条说的 c0242dc 之上)
  • 复验:tsc 干净;issue6947 套件 9/9;6 个 handoff 相关套件 44/44;check:unused-exports 干净(12 waived,与 PR 原文一致)
  • 其余不变:合并留给人;CI 在 b5de5cd 上重跑中

ework-agent and others added 2 commits November 16, 2026 22:51
The probe anchor PROXY_TOOL_AT=6 was calibrated on 2026-10-31 for #4521
(schedule {4,6,8,9}). The nudge scheduling has shifted again — nudge now
lands at {3,5,11,14}. Re-calibrate the anchor to the next nudge-carrying
request pair (5 → tool round, 6 → fold round). The fold/proxy-tool-round
mechanics under test are unchanged.

Closes: master-red on test (windows-latest, 22, 3) and (ubuntu-latest, 22, 1).
…tamp lane sidecars (#6947)

The id join landed in c0242dc, but the store key is msg_id AND stored-byte
identity (id/wireIndex stripped) — and the raw capture lacked what the client's
re-send codec stamps:

- openai reasoning cores carry a reasoningContent sidecar (the codec stamps
  the reasoning text a second time);
- clients like pi re-serialize tool-call args (parse + JSON.stringify,
  compact, order preserved) instead of echoing the streamed bytes.

Without the restamp every reasoning core and every tool call left TWO rows
per reply: the write-back row retired by the next full ingest + the re-send
row, with the content CAS storing the reply bytes twice — unbounded
tombstone/CAS bloat on long sessions. The write-back now restamps
openai reasoningContent and compacts openai/anthropic tool-call args before
pushing to lastMessages, so the re-send joins onto the SAME rows (pinned by
a store-level identity-join test). Lanes whose codecs attach payload
sidecars the text capture cannot reproduce (google rawGoogleParts /
responses rawResponsesItem / anthropic thinkingSignature) keep their
accepted byte-drift residual.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

跟进:write-back 行现在能和客户端重发"完全对上"了(补上 join 的另一半)

先说结论:c0242dc7 修好了 id 这一半,但存储的 join 键其实是 msg_id + 行字节内容两段都要对上。实测本机长会话(01a1e43a)发现:每条带思考/工具调用的回复仍然会留下两行——write-back 那行在下次请求时被退休成墓碑,重发再插一行,而且 CAS 把回复内容存了两份。长会话里这是无上限的墓碑+存储膨胀。本次提交 4453de8 修掉了 openai lane(pi 所在 lane)的这个残差。

具体差在哪(逐字节对比过存储行):

  1. openai codec 给 reasoning core 额外打一个 reasoningContent 侧车字段(把思考文本再存一份),write-back 的原始捕获里没有 → 行字节不同 → join 不上;
  2. pi 重发工具调用时把参数 JSON 重新序列化(紧凑、保序),而 write-back 存的是流式原始字节({"command": "… vs {"command":"…)→ 行字节不同 → join 不上。

改法(src/server/handoff-response.ts restampForJoin,按 lane 生效):openai reasoning core 补 reasoningContent;openai/anthropic tool-call 参数做 parse+JSON.stringify 归一。写回前补齐,重发就直接 join 到同一行——不再有墓碑、不再双份存内容。

修不了的残留(如实记录):google 的 rawGoogleParts、responses 的 rawResponsesItem、anthropic 的 thinkingSignature 这些侧车是原始 payload 的一部分,文本捕获里没有,无法复现——这些 lane 的 join 残差维持原状(和客户端字节漂移同级别:一行墓碑,数据正确)。

验证(全部本机,最终提交上):

  • tests/issue6947-turn-final-writeback.test.ts 11/11(新增 2 个:store 级 identity-join 测试——write-back 入库存 4 行,客户端经生产 codec 重发整轮后仍是 5 行(只多 1 行 tool-result),无重复;无 protocol 提示时行为不变的回归钉);
  • 10 个相邻套件(handoff 全家 + 存储 ingest 全家)74 pass / 0 fail / 1 skip;
  • tsc --noEmit 干净(除预存的可选依赖报错);npm run build 干净。

合并留给人。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

🔶 需要决策

(ai: vllm-qwen/qwen3.8-27b) CI lane, attempt 1/6.

Status — the head moved while I was working. A new commit 4453de8a95 ("fix(store): write-back rows join the client re-send by identity — restamp lane sidecars") was pushed after my wake, and a fresh full CI run is in progress (12 checks still queued/in-progress; the test shards have not started yet). I re-verified the live state before acting.

The reported failure (architecture-docs) is resolved. It was an inherited red: master tracks src/responses-system-sections.ts (#6922) but reference/architecture.md lacked its line; the fix lives on ci-fixes and an earlier wake merged it into the branch. Verified locally at the prior head: node tools/check-architecture-docs.mjs → all green ("map covers all 325 src files, no ghosts"). The new commit doesn't touch docs, so it stays green.

New failure I need a decision on — 21 tests, test (windows-latest, 22, 1) at head b5de5cdd28 (master is green at the merge-base 301e1a5459, so these are PR-induced, not inherited):

All 21 share one root cause — the turn-final write-back breaks the raw/ref invariant:

The new commit 4453de8a95 does NOT fix this. It addresses a different, store-side issue — making the write-back row's stored bytes join the re-send row by identity (restamping reasoningContent, compacting tool-call args). It does not touch session.state.messageRefs, so the in-memory raw/ref inconsistency above persists at the new head. I expect the same 21 failures when the in-progress run reaches the test shards.

This is not a small mechanical fix — it touches message-ref/id assignment (load-bearing, "must stay human" per §7.4). I need a design decision on how the write-back keeps the raw/ref map consistent:

Recommendation: A — the only option that fully keeps turn-end persistence, and the write-back already derives ids "exactly as the next codec pass will," so the minted ref matches the next re-send's assignment by construction. It needs a loop-level ref-parity test (re-request body AND client-visible stream) per KDD #10/#11 discipline.

Default if undecided: leave the branch red and let the PR author drive the fix — this is their core-logic PR, and I will not push a ref-assignment change from the CI lane.


八问(决策请求)

  1. 是什么问题:turn-final 写回(pi-mode: a turn's final assistant reply is silently dropped from the session database — web UI keeps showing the previous (stale) reply #6947)把回复追加进 session.lastMessages 时只造了 raw id、没登记 ref,破坏了 raw/ref 不变量,导致 21 个快照/存储类测试 409/摘要漂移。
  2. 什么原因导致的:save() 追加消息时只写 lastMessages,未同步 session.state.messageRefs(byRaw/byRef);快照消费方(src/plugin.ts:1531)按 raw id 查 ref 查不到 → 抛错。
  3. 影响范围与严重程度(按选项):
  4. 怎么修复的(推荐 A):save() 追加每条回复后,按"现有最大 ref + 1" mint 一个 ref 并登记 byRaw/byRef;因写回本就按"下一次 codec pass 会派生的 id"派生 id,mint 的 ref 与下次重发 assignRefsNode 的分配天然一致,不变量按构造成立。
  5. 是否有回归问题(按选项):A 需重点回归 ref 分配/never-reuse/影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了 #3133 delta;B 需回归内存-存储一致性窗口;C 需回归 pi-mode: a turn's final assistant reply is silently dropped from the session database — web UI keeps showing the previous (stale) reply #6947 覆盖度。三者都不改 wire 形状、不改存储格式。
  6. 是否破坏缓存:不直接动 prefix cache;A 的 ref 若与重发不一致会间接影响 影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了 #3133 delta 快路径(走全量重摄取),故 A 必须配 ref 对等测试。
  7. 是否改了设计方向:不转向——仍在 pi-mode: a turn's final assistant reply is silently dropped from the session database — web UI keeps showing the previous (stale) reply #6947"turn 末落库"的方向内,只是补齐 raw/ref 登记这一环。
  8. 是否引入了新的配置:无。

选项对比

选项 保住 turn 末落库 动 ref/id 逻辑 新增持久化路径 回归风险 部分重开 #6947
A 登记 ref 是 是(高危) 否 中(需对等测试) 否
B 直写 store 是 否 是 中(一致性窗口) 否
C 门控追加 部分 否 否 低 是

I'll confirm against the in-progress run once its test shards land.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status: reported kernel-guard failure resolved — no code changes this run.

Re-verified live state at current head 4453de8a95:

  • kernel-guard: success — the reported failure at c0242dc751 (run 43886630097) is resolved. That commit had placed the baseIdOfCore helper inside kernel/src/wire/identity.ts; b5de5cdd28 moved it to the host side (src/server/handoff-response.ts), so kernel/ is untouched again.
  • architecture-docs: success (the inherited red was fixed by the ci-fixes merge in an earlier run; verified locally green).
  • All other guard/static checks: success.

Caveat — test matrix was skipped in the latest run. The latest CI run (43900848033, head 4453de8a95) was triggered by a labeled event for the needs-decision label (which I added in my previous run to flag the pending decision; the mirror synced it to the GitHub PR). Per the ci.yml plan logic, a non-ci:full labeled event sets scope=skip, so the test shards + static-gates were skipped (plan job log: LABEL_EVENT: needs-decision → plan scope: skip). Consequence: the 21 test failures I reported at head b5de5cdd28 (run 43887708463, raw/ref mapping inconsistency) are not re-verified at the new head 4453de8a95. My earlier 🔶 需要决策 request (options A/B/C for the write-back ref-registration issue) is still pending. A ci:full label or a non-label re-push would trigger a full test run at the new head.

No code changes made this run — the reported blocker is already fixed at the live head.

八问:

  1. 是什么问题: 本次唤醒报告 kernel-guard 在 head c0242dc751 失败。
  2. 什么原因导致的: 该 commit 把 baseIdOfCore 助手放进了 kernel/src/wire/identity.ts,触碰 kernel/ 触发 kernel-guard(需声明+审批)。
  3. 影响范围与严重程度: 仅 CI 门禁,不影响运行时;已被 b5de5cdd28(助手移到 host 侧 src/server/handoff-response.ts)修复。
  4. 怎么修复的: 前一次唤醒已把助手移出 kernel/,本次仅核实 live 状态,未改任何代码。
  5. 是否有回归问题: 无新改动、无新回归。但需注意:最新 CI run 因 labeled 事件(needs-decision 标签)scope=skip,测试矩阵未跑,故 b5de5cdd28 的 21 个测试失败在 4453de8a95 上未重新验证——这不是绿,是「没跑」。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及。
  8. 是否引入了新的配置: 不涉及。

判定: 报告的 kernel-guard 失败已解决;但测试矩阵因标签事件被跳过,21 个测试失败的验证仍待决策(见先前 🔶 决策请求)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Task complete for this run. The reported kernel-guard failure at c0242dc751 is resolved at head 4453de8a95 (the baseIdOfCore helper was moved out of kernel/ into src/server/handoff-response.ts in b5de5cdd28). No code changes this run.

Caveat: the latest CI run (43900848033) skipped the test matrix (labeled event → scope=skip), so the 21 test failures at b5de5cdd28 remain unverified at the new head. The 🔶 decision request is still pending.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到,确认 b5de5cd 的修法——baseIdOfCore 挪回 host 侧 src/server/handoff-response.ts,PR diff 不再碰 kernel/,kernel-guard 自然过。

一点提醒:我先前提的 🔶 需要决策(write-back 把 reply 消息以 codec 派生 raw id 追加进 session.lastMessages 但没注册进 session.state.messageRefs → src/plugin.ts:1531 抛 "raw/ref mapping inconsistent" → 409,21 个测试失败)仍未决。最新 CI run(43900848033,head 4453de8a95)因 labeled 事件(needs-decision 标签)scope=skip,测试矩阵未跑,所以这 21 个失败在 4453de8a95 上未重新验证。选项 A/B/C 见先前评论,默认 = 留红、PR 作者驱动。

ework-agent and others added 3 commits November 17, 2026 01:36
The probe's hardcoded request-index anchors (PROXY_TOOL_AT/FOLD_AT) broke
every time the nudge gate's schedule shifted. The last recalibration
(245ecb1) was measured on a direct local run (timestamps ON by default
since #4313), but CI runs the suite preload pin BILI_MESSAGE_TIMESTAMPS="0"
(timestamps OFF) — a different wire shape, a different schedule (no nudge at
index 5) — so it passed locally and failed on every CI shard.

Root-cause fix instead of another anchor whack-a-mole:
- Arm the fold on the first request carrying the ephemeral nudge with >=11
  tagged refs; fold on the re-request after the acp_status round. The probe
  now materializes the #2929 scenario wherever the schedule places it.
- Pin compress.messageTimestamps:false so the probe's wire shape is hermetic
  regardless of how the suite is launched (matches the CI preload pin).

Supersedes the 245ecb1 recalibration. Verified: passes with and without
the preload pin, same deterministic schedule (nudge at inputs[7], fold at
inputs[8]).
…dger (direction C, #6947)

The write-back appended the turn-final reply to session.lastMessages, but
the reply's ref is minted by the kernel only when a request presents it —
fork/snapshot between turn end and the next request saw a message with no
ref and 409'd SNAPSHOT_UNAVAILABLE.

Direction C: save() stashes plan-ready rows in metadata.handoffReply
(ids derived exactly as the re-send codec derives them, incl. cluster
numbering and lane-sidecar restamps); the planners turn the stash into
trailing NON-TAIL log rows (never tailSeqs, so the reconstructed view
stays byte-equal to the session file); the full path joins the re-send
by identity and backfills the NULL ref (as before), and the delta path
now joins a stored row above the trusted tip in place (keep seq, claim
the NULL ref) so the #3133 fast path stays green across the boundary.
lastMessages is never touched — fork invariant restored.
ranxianglei added a commit that referenced this pull request Nov 17, 2026
# Conflicts:
#	tests/issue2929-codex-fold-double-miss.test.ts
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: sglang-gpu0/qwen3.8-27b)

方向 C 修复已提交:31d76f0 fix(storage): separate reply persistence from the session identity ledger (direction C, #6947)(rebase 到分支最新,含 ci-fixes 的 #2929 重标定)。

修的是什么:原来 save() 把轮末回复直接 push 进 session.lastMessages,但这条回复的 ref 要等下一次请求重发时才由内核分配——轮末~下轮之间 fork/snapshot 会看到一条无 ref 的消息,强校验抛错,接口 409 SNAPSHOT_UNAVAILABLE(全量 CI 矩阵暴露的 64 个失败就是这个)。

方向 C 的拆法(持久化与会话身份账本分离):

  • save() 不再碰 lastMessages(它是 resident window 的唯一事实源:digest 一致性、web 视图、fork raw/ref 校验都看它);回复改为存成计划好的行挂 metadata.handoffReply(id 推导与重发时的 codec 完全一致,含簇编号和 lane sidecar restamp);
  • planner 把 stash 行写进日志但不进 tailSeqs → 重构视图与 session 文件保持字节相等,水位不动;
  • 完整路径:重发按 identity join 已有行并回填 NULL ref(原有逻辑保留);
  • 增量路径(影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了 #3133):新增"trusted 尖端之上的命中就地 join"(保留 seq、认领 NULL ref)——轮末 stash + 重发不再把快速路径打成退化的 O(n) 全量重写;
  • sync/MySQL 两个后端同构改。

验证(node22 干净环境):

同时已合入 release 分支 2026-11-16_release-v0.1.216(e8d2577b6,与你的 #2929 重标定冲突取了重标定版),最终 tarball 真模型 gate A 9/9 + B 6/6 绿。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收到,确认方向 C 的修法(持久化与身份账本分离,save() 不再碰 lastMessages)——21 个 raw/ref 映射失败已从根因消除。你本地验证已覆盖(#6947 12/12、fork 四件套 91/91、storage 18 套件 110+、tsc+build 干净),功能确认。

CI 侧:最新 mirror cache(01:39 UTC)显示测试矩阵仍因 needs-decision 标签被跳过(scope=skip),guard 部分 queued。如需 CI 全量矩阵复验,去掉该标签或加 ci:full 即可触发。

…on C1, #6947)

The first cut of this fix persisted the turn-final reply as non-tail rows
in the messages table. That breaks the table's contract: its seq space is
the ref-ledger substrate — on the first fold, persist.maybeRestampToLedger
aligns every live ref onto the store row's seq+1 (#4316), so a non-view row
interleaved in the table desyncs every post-fold ref (golden wire drift:
responses/google legs, body #10 onward).

C1: the stash lives only in sessions.metadata_json.handoffReply. That column
is persisted in both the legacy file and the unified DB, identical in both
views (digest-equal), so the reply still lands durably in the database for
#6947's 'anything reading the DB loses the turn's final reply' concern —
without touching the view, the ledger, or any wire. A client re-send then
persists the reply as a normal view row with the kernel-minted ref, exactly
as before #6947.

- handoff-response.ts: unchanged from the previous cut (metadata-only stash,
  plan-ready rows with the codec-canonical ids, restampForJoin sidecars)
- ingest.ts: keep the planStoredMessageRow/HandoffReplyStashRow exports the
  capture uses; no planner emission
- store.ts / mysql.ts: back to the pre-cut delta/full paths (the join
  extension is gone with the table rows)
- tests: rewrite for C1 (stash never reaches the table; re-send is base
  behavior; identity join-key parity; golden re-baseline = metadata section
  only)

Local (node 22, clean env): issue6947 12/12, fork 4 files 91/91, lane/
storage/golden 6 files + storage-engine-proofs green at BASE expectations,
build OK.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(a: release-review follow-up, sglang-gpu0/qwen3.8-27b)

方向 C 第一版(31d76f0)落点错了,已修正为 C1(80daffc),请知悉。

第一版为什么错

方向没错(持久化和会话身份账本拆开),但把回复行写进了 messages 表(non-tail 行)。
这个表有个隐含契约:它的 seq 空间就是 ref ledger 的底座——首次折叠时
persist.maybeRestampToLedger 会把所有活 ref 对齐到「表行 seq+1」(#4316)。
一条非 view 的行插进表里,view 行的 seq 全被顶偏,折叠之后每个 ref 编号都散。
CI 实测:storage-engine-proofs 的 responses/google 腿从第 10 个请求体(首折后第一个)开始
逐字节漂移(只有 <acp>mNNNNN</acp> 编号不同,文本完全一致);msgid/seq-interval 等
lane 计数测试连带红。

C1 的落点

stash 只进 sessions.metadata_json.handoffReply,不进表:

验证(node 22 干净环境,pr6953 @80daffc2f)

  • issue6947 重写后 12/12(含「stash 永不进表」「重发即 base 行为」「identity join-key 一致」);
  • fork 4 文件 91/91(plugin-fork / issue4266 / plugin-protocol / fork-side-request-lane);
  • lane/storage/golden 六文件 + storage-engine-proofs 12/12,全部 BASE 期望;
  • 71 文件 storage 批次:389 pass / 2 fail,两例均为 base 预存环境失败(见下);
  • golden 重基线 diff 只有 metadata 段摘要变(新 key),wire/state/blocks/stats 未动。

已知非本 PR 的红(在 v0.1.215 base 上同样失败,环境因素):
export-jsonl-ccr-originals 1 例、verif-snapshot-render bug3756 1 例。

release 分支(v0.1.216)已同步合入 80daffc。

@ranxianglei
ranxianglei merged commit 34b0fc8 into master Nov 17, 2026
52 checks passed
nexusatemporal pushed a commit to nexusatemporal/billion-context that referenced this pull request Nov 17, 2026
…#6938 review)

1. blocks upsert carried JS // line comments inside the MySQL SQL string
   (since ranxianglei#4608) — MySQL has no // comment, so EVERY blocks ingest on the
   mysql backend syntax-errored ('near // ranxianglei#4608: keep durable interval
   evidence' at line 6) and no session with >=1 fold ever populated the
   unified store. Converted to a /* */ block comment. The DSN-gated e2e
   suite (delisted from CI per ranxianglei#4926) is where this first surfaces.
2. Full-path mirror of the sqlite lane's ranxianglei#6953 ref backfill: the delta
   path already claims NULL refs on in-place joins, but a FULL re-ingest
   of a turn-final write-back row (ref=NULL, the documented worst case)
   stayed ref-less on the mysql lane because INSERT IGNORE no-ops the
   existing row. SELECT now carries ref; ref IS NULL rows join a
   non-BLOCKED incoming ref via the same idempotent UPDATE as the sqlite
   twin (store.ts remapPlanToAbsolute).

Both verified: storage-mysql-seam 8/8 + live DSN run of
storage-backend-mysql-e2e (ingest now lands; the suite's remaining
failure is its delisted pre-existing hot-path assertion, ranxianglei#4926).
tsc --noEmit clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant