Skip to content

[PR] #4316 Layer 3 (number directly mapped to canonical seq) PR 1/3: Fold-overwrite into real seq range | #4316 层3(编号直连正史 seq)PR 1/3:折叠覆盖写入真实 seq 区间 - #4335

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-22_4316-layer3-seq-bounds
Oct 24, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-22_4316-layer3-seq-bounds

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

背景

#4316 三层定案的第 3 层(owner 拍板 2026-10-22):折叠记账从 h_id 集合迁到正史 [start_seq, end_seq] 区间。本 PR 是该层的 PR 1/3,只做存储侧「填 seq 边界」——为后续 reconcile 区间证据层(PR 2)与内核编号直连(PR 3)打地基。

改动

  • blocks.start_seq/end_seq(两列早存在、NOT NULL、此前恒为占位 0)现在 ingest 时从 append-only 正史解析出真实值:块覆盖消息 id → 其 seq 的 min/max。
  • 实现放在 ingestSession(src/storage/store.ts sqlite + src/storage/mysql.ts 双后端),而非纯 planner——最终绝对 seq 在事务内重编(delta 占位 seq 于事务里 renumber),planner 拿不到。
  • 覆盖 id 在正史无行(一次性导入的存量折叠块)→ 保持 0 占位,诚实缺口,绝不伪造边界。

行为变更披露

start_seq/end_seq 从「恒 0」变为「可解析时 = 覆盖消息真实 seq 区间」。当前无任何压缩路径读取这两列,仅 export.ts 诊断面会看到真实值(0 → 真值,属诊断修正非回归)。压缩行为零变化。无 schema 迁移(列本就存在)、无 kernel 改动(纯 host src/storage)、无新增配置/env/CLI。

验证

  • 定向:tests/issue4316-seq-bounds.test.ts(红桩→绿)+ 整个 storage-* 测试簇 28/28 通过。
  • npm run typecheck 通过。
  • 全量回归交给远程 CI(push 后看 test 车道)。

Part of #4316(layer 3, PR 1/3)。

awork added 2 commits October 23, 2026 00:52
…ipt coverage

Pins the storage-side contract for "编号直连正史 seq": a block whose covered
messages exist as append-only rows carries their real min/max seq; unresolvable
coverage (imported pre-fold) keeps the 0 placeholder. Red against the current
hard-wired 0 at the single write site (src/storage/ingest.ts).
…ranscript

Resolve each block's covered ids to canonical [start_seq, end_seq] bounds at
ingest time (store.ts sqlite + mysql.ts), after final absolute seqs are known
(delta placeholders renumber in-transaction, so the pure planner cannot compute
them). Unresolvable coverage keeps the 0 placeholder — honest gap, never a
fabricated bound. Zero compression-behavior change: nothing reads these columns
yet; only export surfaces them (diagnostic 0 -> real).
@ranxianglei ranxianglei changed the title #4316 层3(编号直连正史 seq)PR 1/3:折叠覆盖写入真实 seq 区间 [PR] #4316 Layer 3 (number directly mapped to canonical seq) PR 1/3: Fold-overwrite into real seq range | #4316 层3(编号直连正史 seq)PR 1/3:折叠覆盖写入真实 seq 区间 Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-22_4316-layer3-seq-bounds (05dc683)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5141-pr.4335.7307a36

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4335.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 这是父 issue #4316「切模型/客户端重写历史后消息身份全换血」三层定案的层3 PR 1/3;近期无其他 PR 在填 blocks.start_seq/end_seq。相关但不同:#4334(把 storage-durability.test.ts 注册进 LIVE_FILES——正是当前共用红门禁的来源,机制无关)、#4307(回执与覆盖判定,#4316 引用的相邻机制)。

标签: bug + scope/storage + p3(理由:这是修 #4316 缺陷的地基步,落在存储层;本 PR 自身无用户可见行为变化,故 p3)。已打。


我把 diff、双后端上下文、planner、列的读者、以及 PR head 与 master 两边的 CI 都过了一遍。结论:代码正确、范围克制、披露完整、方向对,有一处性能点建议合入前顺手改掉。逐问如下。

1. 是什么问题 — #4316 层3 要"折叠记账从覆盖的消息 h_id 集合迁到正史 [start_seq,end_seq] 区间"。本 PR 只做第一步地基:ingest 时给 blocks.start_seq/end_seq(两列早存在、NOT NULL、此前恒写占位 0)填上真实值,为 PR2(reconcile 区间证据层)和 PR3(内核编号直连)铺路。

2. 什么原因导致的 — "恒 0"的根因是 planner(src/storage/ingest.ts:349-350)在事务外拿不到最终绝对 seq:delta 占位 seq 要在 ingestSession 事务内 renumber 才定案,planner 阶段只能写 0。所以修复放在唯一能拿到真 seq 的位置——写块那一步,用块覆盖的 msg_id join 正史 messages.seq 取 min/max。方向对:修的是"值在哪一层才确定",不是把 0 换个地方糊上去。

3. 影响范围与严重程度 — 当前没有任何压缩路径读这两列(grep 全仓确认),只有 src/export.ts:259-260 诊断面会看到(0→真值,属诊断修正非回归)。压缩行为零变化、无 schema 迁移、无 kernel 改动、无新配置/env/CLI。属"仅内部实现、暂无外部影响"的地基步。真正风险不在功能而在性能(见第5问)。

4. 怎么修复的 — sqlite(store.ts)+ mysql 对称改法:ingestSession 里先按 session 建 msg_id→[minSeq,maxSeq] 映射,再对每个块解析 payloadJson 里的 effectiveMessageIds+directMessageIds,取覆盖 id 的 seq min/max 写入;覆盖 id 在正史无行(存量导入的旧折叠块)或 payload 坏 → 保持 [0,0] 诚实缺口,不伪造边界。覆盖字段口径与既有 coveredHash(ingest.ts:345,359)完全一致(都用 effective+direct、不含 directBlockIds)——这是个好的自洽信号。mysql 侧 Number(r.seq) 对 BIGINT 列的处理也正确。

5. 是否有回归问题(重点)—

  • 既有行为逐项:两列从"恒 0"→"可解析=真实区间",已披露且无读者依赖旧 0,非回归。fail-fast 未变(坏 payload catch 后保持占位,合理);阈值/时序/输出格式无变化。
  • ⚠️ 性能回归风险(建议合入前处理):每次 ingestSession(=每次会话持久化,shadow.ts:330/437/490/792/800/876)只要存在块就 SELECT msg_id, seq FROM messages WHERE session_id=? 拉全表建 Map。项目主打的是亿级 token / 上万轮的马拉松会话,messages 表很大——这是新增的每次持久化 O(全 transcript)扫描 + 内存分配。数据今天只作诊断用所以不是可用性阻断,但对旗舰场景是实打实的成本。具体修法(几行):先收集所有块覆盖 id 的并集,再 … AND msg_id IN (<并集>) 只取需要的行(单次折叠覆盖的 id 数远小于全表)。
  • 小瑕疵:seq 是 0 起编(store.ts:764 nextSeq=0; abs=nextSeq++),hi<0 ? [0,0] 这个哨兵会让"只覆盖 seq=0 那条消息"的真实折叠也落成 [0,0],和"不可解析"混淆。极端边缘,但用 lo===Number.POSITIVE_INFINITY 判空更干净。

6. 是否破坏缓存 — 不涉及。纯存储侧记账列填充,不碰前缀缓存/重试/增量游标/去重键。

7. 是否改了设计方向 — 不转向,是既定三层方案的落地第一步,边界清晰(host src/storage,不动 kernel)。

8. 是否引入新配置 — 无。无 env/config/CLI/默认值变化。


测试有效性 — 两个用例都有牙:正向断言 startSeq>0 且等于覆盖首/末消息的 seq、endSeq>=startSeq;负向断言覆盖 id 缺失时保持 [0,0] 且不崩。缺口(不阻断,建议补):directMessageIds 分支未单测(用例只走 effective)、部分可解析的混合场景未测、seq=0 哨兵边界未钉。mysql 路径此单测不覆盖(sqlite only),靠 CI mysql-contract 车道兜底。

CI 状态澄清(关键) — PR head 05dc68319642 上 test 车道 12 个 shard + mysql-contract 全红,但这不是本 PR 造成的:master tip aa198fe613b4 同样全红、签名完全一致——test shard 挂在 scripts/run-ci-tests.mjs 的 LIVE_FILES 门禁(tests/storage-durability.test.ts 未注册,由独立 PR #4334 修),mysql-contract 在 master 上也红(该 lane 日志端点对本凭据返回 404,拿不到具体断言行,但 master 基线同样红足以排除本 diff 引入)。也就是说整个仓库当前处于共用红状态,得等 #4334 落地(或 rebase 到其后)本 PR 才能显示绿;本 PR 自身 diff 不引入这些失败。

diff 整洁度 / 代码质量 — 双后端对称镜像符合本文件既有惯例(store.ts/mysql.ts 本就处处互镜);命名达意(seqSpanById/boundsForCovered);错误处理完整(JSON.parse try/catch 带注释说明保持占位);无死代码、无魔法数。可议处:seqSpanById+boundsForCovered 这段在两后端重复(≥2 处),但 sqlite 同步/mysql 异步 idiom 不同,强抽公共会牺牲可读性——遵循现有镜像惯例可接受;若要抽,只抽"给定 rows 数组算 bounds"的纯函数部分即可。


合并判定:LGTM(带一个建议项)。 代码正确、范围克制、披露到位、方向对。建议合入前:①把全表扫描改成 IN(覆盖id并集)(性能,几行);②可选补 directMessageIds/混合/seq=0 三个测试缺口 + sentinel 用 Infinity 判空。以上都非阻断项——真正的合入门槛是等 master 的 #4334 门禁清了让 test 车道转绿(本 PR 自身无责)。

性能那条要不要我直接在这个分支上补 commit?因为是你自己按"PR 1/3 只做填边界"刻意收着的最小范围,我没擅自动你的分支——你点头我就推(独立标注 commit);或者留到 PR2 开始真正读这两列时一并优化也行。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.793-master.8735ac4 (rolling master tag).

npm install -g billion-context@0.0.793-master.8735ac4  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
)

#4335's block start_seq/end_seq fill ran `SELECT msg_id, seq FROM messages
WHERE session_id=?` on EVERY persist whenever blocks existed — an O(full
transcript) scan plus full-table Map allocation, real cost on the flagship
marathon sessions where messages is huge.

Collect the union of every block's effective+direct covered ids first, then
fetch only those rows via a chunked `msg_id IN (...)` (500/batch). The
(session_id, msg_id) index makes each batch targeted lookups; chunking keeps a
heavily-folded session under the driver's bind-parameter cap so an over-limit
query cannot throw and feed record()'s 3-strike self-disable. The min/max span
accumulation and [0,0] placeholder semantics are unchanged — restricting the map
to covered rows is byte-equivalent because boundsForCovered only ever looks up
covered ids. Applied identically to the sqlite (store.ts) and mysql.ts twins.

Adds tests/issue4512-ingest-span-in-clause.test.ts pinning the two paths the
old code never exercised: a covered-id union spanning multiple query batches,
and directMessageIds feeding the union. Existing #4316 seq-bounds regression
tests still pass.
ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
)

#4335's block start_seq/end_seq fill ran `SELECT msg_id, seq FROM messages
WHERE session_id=?` on EVERY persist whenever blocks existed — an O(full
transcript) scan plus full-table Map allocation, real cost on the flagship
marathon sessions where messages is huge.

Collect the union of every block's effective+direct covered ids first, then
fetch only those rows via a chunked `msg_id IN (...)` (500/batch). The
(session_id, msg_id) index makes each batch targeted lookups; chunking keeps a
heavily-folded session under the driver's bind-parameter cap so an over-limit
query cannot throw and feed record()'s 3-strike self-disable. The min/max span
accumulation and [0,0] placeholder semantics are unchanged — restricting the map
to covered rows is byte-equivalent because boundsForCovered only ever looks up
covered ids. Applied identically to the sqlite (store.ts) and mysql.ts twins.

Adds tests/issue4512-ingest-span-in-clause.test.ts pinning the two paths the
old code never exercised: a covered-id union spanning multiple query batches,
and directMessageIds feeding the union. Existing #4316 seq-bounds regression
tests still pass.
ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
)

#4335's block start_seq/end_seq fill ran `SELECT msg_id, seq FROM messages
WHERE session_id=?` on EVERY persist whenever blocks existed — an O(full
transcript) scan plus full-table Map allocation, real cost on the flagship
marathon sessions where messages is huge.

Collect the union of every block's effective+direct covered ids first, then
fetch only those rows via a chunked `msg_id IN (...)` (500/batch). The
(session_id, msg_id) index makes each batch targeted lookups; chunking keeps a
heavily-folded session under the driver's bind-parameter cap so an over-limit
query cannot throw and feed record()'s 3-strike self-disable. The min/max span
accumulation and [0,0] placeholder semantics are unchanged — restricting the map
to covered rows is byte-equivalent because boundsForCovered only ever looks up
covered ids. Applied identically to the sqlite (store.ts) and mysql.ts twins.

Adds tests/issue4512-ingest-span-in-clause.test.ts pinning the two paths the
old code never exercised: a covered-id union spanning multiple query batches,
and directMessageIds feeding the union. Existing #4316 seq-bounds regression
tests still pass.
ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
)

#4335's block start_seq/end_seq fill ran `SELECT msg_id, seq FROM messages
WHERE session_id=?` on EVERY persist whenever blocks existed — an O(full
transcript) scan plus full-table Map allocation, real cost on the flagship
marathon sessions where messages is huge.

Collect the union of every block's effective+direct covered ids first, then
fetch only those rows via a chunked `msg_id IN (...)` (500/batch). The
(session_id, msg_id) index makes each batch targeted lookups; chunking keeps a
heavily-folded session under the driver's bind-parameter cap so an over-limit
query cannot throw and feed record()'s 3-strike self-disable. The min/max span
accumulation and [0,0] placeholder semantics are unchanged — restricting the map
to covered rows is byte-equivalent because boundsForCovered only ever looks up
covered ids. Applied identically to the sqlite (store.ts) and mysql.ts twins.

Adds tests/issue4512-ingest-span-in-clause.test.ts pinning the two paths the
old code never exercised: a covered-id union spanning multiple query batches,
and directMessageIds feeding the union. Existing #4316 seq-bounds regression
tests still pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant