Skip to content

[PR] Review P2 fixes: dedupe the double body scan, pin expiring media refs, snapshot pre-OOM envelopes before overwrite, trace msgid-rescue - #5895

Merged
ranxianglei merged 2 commits into
masterfrom
fix/p2-review-batch-5865
Nov 7, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
fix/p2-review-batch-5865

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

这是什么问题

#5865 大 diff review 定级的 4 个中等问题(P2:不立刻坏,但都是实打实的浪费或隐患):

  1. 同一请求体被整包扫描两遍:入口的 early-hold 预检闸门(src/server/handle.ts)和转发前的正式 429 闸门(src/server/relay.ts)各自独立扫同一个几 MB 的 body 找顶层 model 字段——每 MB 约 1ms,纯重复劳动,每个代理 POST 都付一次。
  2. 折叠判定把「带签名的临时 URL」当成可恢复引用([Bug] Config save rejects valid mitm:// provider lanes as "invalid recipe on named provider" | 整文件保存把 mitm:// 路由误判为命名配方 #5620 的漏洞回潮):S3/GCS/Azure 预签名链接、CDN session token 几小时后就失效,而折叠摘要恰恰是几小时后才解压——折叠时把媒体字节扔了,回头 URL 一死,载荷就永久丢了。[Bug] Config save rejects valid mitm:// provider lanes as "invalid recipe on named provider" | 整文件保存把 mitm:// 路由误判为命名配方 #5620 之前老行为是永久 pin 住,反而更安全。
  3. OOM 解码失败 → 静默覆盖丢档:会话文件在内存压力下解码 OOM(字节健康、只是这轮读不动)会恢复成全新会话;它的第一次保存直接原子覆盖旧档——没有取证副本、没有日志,静默永久丢失。
  4. msgid-rescue([resolveSessionOrder] Full compaction and persona tag drift occurring in the same turn → inevitably forks to |sub: side | [resolveSessionOrder] Full compaction and persona tag drift occurring in the same turn → inevitably forks to |sub: side | [resolve… #5719 兜底路径)零日志:每次触发无声无息,「阈值 ≥1 还是 ≥2」的决策一直没有实地证据。

怎么修的

  1. requestModelFromBody 增加请求级 memo(WeakMap 挂在 IncomingMessage 上,body 按 identity 锁定):两个闸门共享同一次扫描;body 变了自动重扫。新增测试计数器 pin「一次扫描、两处消费」+ 4 个单测。
  2. kernel media-refs:X-Amz-*/X-Goog-*/X-MS-*/SAS/Expires/Signature/token 等签名/过期特征的 HTTP(S) URL 一律视为不可恢复 → 消息继续 pin。宁可误 pin 一条消息,不可丢载荷。11 个新用例(S3 v4/v1、GCS、Azure SAS、CDN token、以及 durable URL 不误伤)。
  3. persist:OOM 文件(id 记入 preoomPending)在下次覆盖保存前先 rename 成 .corrupt-preoom-<ts> 快照(沿用 .corrupt- 家族的 cap-1 清理);保存失败把快照改回去,绝不把健康字节困在取证名后面。
  4. session-resolve:msgid-rescue 每次触发打一行 warn(hits/exit 上下文/首命中 id)。

kernel-change: media-refs classifies pre-signed/expiring HTTP(S) URLs (X-Amz-/X-Goog-/SAS/Expires/Signature/token) as NON-recoverable — messages keep pinning instead of folding out bytes whose only handle lapses; kernel 0.0.185 → 0.0.186, lockfile synced

评审里另一条「loadConfigFile 每次调用 readFileSync」不改:#2078 已把「内容为准、mtime/size 不可信」钉成测试(防 web-UI Apply 在粗时间戳文件系统上读到旧值),stat 门会破坏这个安全语义,换来的只是每请求几十微秒——收益配不上风险。armInternal 的两次 resolve 也是刻意的(第一次挡掉多 MB body 的 JSON.parse)。

验证

ework-agent added 2 commits November 7, 2026 23:06
…hot, msgid-rescue trace

- throttle F1: requestModelFromBody memoizes {body→model} on a request-scoped
  cacheHost so the pre-ingest early-hold gate and the full 429 gate share ONE
  body scan instead of walking the multi-MiB document twice per POST
- media-refs F2 (kernel): pre-signed/expiring HTTP(S) URLs (X-Amz-*/X-Goog-*/
  X-MS-*/SAS/Expires/Signature/…) are classified NON-recoverable — the handle
  lapses before a folded summary is decompressed, so the message keeps pinning
- persist: an envelope that OOM'd on decode is snapshotted before the next
  save overwrites it (preoomPending set + withPreoomSnapshot)
- session-resolve: every msgid-rescue firing now logs hits/exit-context as
  #5719 overlap-rung reachability evidence
- kernel 0.0.185 → 0.0.186 (media-refs change), lockfile synced

Review findings from #5865 master-diff review (P2 batch).
…THING, not the bare prefix

The alternation's $ anchor made 'X-Amz-Algorithm' (and every other
x-<provider>-* signed param) fall through as durable. Keep $ only on the
exact-key branch.
@ranxianglei ranxianglei changed the title Review P2 fixes: dedupe the double body scan, pin expiring media refs, snapshot pre-OOM envelopes before overwrite, trace msgid-rescue [PR] Review P2 fixes: dedupe the double body scan, pin expiring media refs, snapshot pre-OOM envelopes before overwrite, trace msgid-rescue Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: fix/p2-review-batch-5865 (178fc03)

Latest build — install this exact version

npm install -g billion-context@0.0.8542-pr.5895.ce93a3b

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
178fc03 billion-context@0.0.8542-pr.5895.ce93a3b build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5895.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei
ranxianglei merged commit cda6e73 into master Nov 7, 2026
52 checks passed
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1076-master.cda6e73 (rolling master tag).

npm install -g billion-context@0.0.1076-master.cda6e73  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei ranxianglei added bug Something isn't working p1 scope/kernel labels Nov 7, 2026
Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Nov 7, 2026
…ch; relay hunk disjoint from the lane's egress

Adopts the three-commit ranxianglei#5895 batch: model-scan memo, ephemeral-ref
pinning, preoom snapshot and msgid-rescue trace, plus the kernel-side
`EPHEMERAL_QUERY_PARAM` prefix fix (`178fc0318`). Kernel is now
`acp-kernel@0.0.186`.

The only shared file is `src/server/relay.ts`. Master's edit is a single
one-line hunk at `912` (`forwardCaptured` bookkeeping); the lane's egress
guard sits at `2062`. Hunk ranges are disjoint, so git auto-merged without
a conflict and both sides verify present afterwards:

- `src/server/relay.ts:2062` — `await pipePluginJson(toClient, res, undefined, prepared.protocol, undefined, undefined, undefined, reply);`
- `src/storage/mysql.ts:77` — `export const MYSQL_SCHEMA_VERSION = 11;`

Verified on the merged tree (CI preload chain, `BILI_STORAGE_UNIFIED=legacy`,
kernel rebuilt):

tests/issue5175-drift-name-leak.test.ts 58/0, tests/tag-echo.test.ts 79/0,
tests/issue2190-degen-close.test.ts 32/0, tests/golden-storage-render.test.ts 5/0,
tests/storage-engine-proofs.test.ts 12/0, tests/wire-contract.test.ts 23/0,
tests/issue4666-migration-dedup.test.ts 2/0,
tests/native-compact-terminal.test.ts 1/0,
tests/issue5865-model-scan-memo.test.ts 4/0,
kernel/tests/issue5620-ref-recoverable-media.test.ts 11/0.

`tsc --noEmit` clean; the four repo gates exit 0; `gen-config-docs generate`
is a no-op on the four generated artifacts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1 scope/kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant