Skip to content

[PR] debug: staged heap probes over the request processing window (--debug gated, zero behavior change) - #3552

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-17_issue3386-debug-memory-probes
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-17_issue3386-debug-memory-probes

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

For #3386 (diagnostic aid — NOT closing it; the real fix awaits the owner's A/B/C decision on per-turn image budget / folding / dedup).

Why

#3386's crash signature — one 25 MiB / 1372-message request driving the V8 heap to the 4 GB default ceiling mid-turn, ~10 s crash-restart loop — cannot be attributed to its allocating stage from outside the process. The heap snapshot pinned the MASS (99.6% strings, 97.9% image-related, base64 present three ways), but which pipeline boundary materializes it per request is exactly what the pending fix options need:

  • A (per-turn image byte cap) needs to know if the mass lands during parse or during the fold/processTurn window;
  • C (raw-base64 vs data-URI dedup) needs to know how much of it is the duplicate copies built by toString+JSON.

What

Three mem-probe samples in src/server/handle.ts, each logging heapUsed / external / rss (+ the stage's payload bytes):

  1. inbound-parsed — after readBody + content-decoding + bodyBuffer.toString("utf8") + JSON.parse, once session identity is resolved. Baseline before the heavy local-window work.
  2. prepared — after runPrepare() returns, on BOTH paths that produce a Prepared (codex-intercept forge path and the main path). Cumulative cost of the incoming-image index build, wire→core conversion, kernel processTurn (fold/reconcile), and the outbound rebuild + JSON.stringify.

Delta between consecutive samples = the cost of the stage in between. Log line shape: [sessionId] mem-probe <stage>: heap=XMiB external=YMiB rss=ZMiB payload=NB — stable prefix for grep continuity.

Guarantees

  • Zero new config surface: gated on the existing --debug flag (same precedent as the parsed-body dump already at handle.ts under opts.debug). No env var, no config field.
  • Zero behavior change when off: memProbe early-returns before touching process.memoryUsage() when debug is false; when on, it only appends debug-level log lines via the existing logger (no console.error).
  • No kernel changes, no wire changes, no timing-sensitive code touched.

Verification

npm run typecheck clean; targeted handle-path suites (dsh-title-sidecar-handle, issue2824-regression, session-continuity-proof, reasoning-echo) 78/78 green; npm run build green (dist-external-zod gate OK). Full-suite verdict comes from CI on this head.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-17_issue3386-debug-memory-probes (982ecf2)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4222-pr.3552.c63fec2

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3552.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

查重: 不重复。 检索了 open+closed 的 issue 与 PR(关键词 mem-probe / heap / memory / OOM):无其他在途或已合并的记忆探针类改动。本 PR 与 #3386 是「诊断辅助 ↔ 待修 bug」关系而非重复——PR 正文已明确 NOT closing #3386,真正的修复等 owner 对 A/B/C 方案的拍板,两者边界清晰。

开始审核:拉取分支 2026-10-17_issue3386-debug-memory-probes(head c3a0281,1 commit,+16/-0,仅 src/server/handle.ts),按八问体例出结论。注意:base d924789 落后当前 master(v0.1.198 release merge b10c525 之后),mergeable=true、无冲突,合入前需 rebase。

awork added 2 commits October 17, 2026 20:20
One oversized request aborting the whole process (#3386: 25 MiB / 1372 msgs → 4 GB heap) cannot be attributed to its allocating stage from outside. Three mem-probe samples, gated on the existing --debug flag (zero new config surface, zero behavior change when off):

- inbound-parsed: after readBody+decode+toString+JSON.parse and session identity resolution — baseline before the heavy local window work
- prepared: after runPrepare returns (both the codex-intercept and the main path) — cumulative cost of image-index build, wire→core conversion, kernel processTurn, and outbound rebuild/stringify

Each line logs heapUsed/external/rss plus the stage's payload bytes, prefixed 'mem-probe <stage>:' for grep continuity. The delta between consecutive samples is what tells the owner whether the OOM mass sits in parse materialization or in the fold/processTurn window — input for the #3386 A/B/C fix decision.
…dies

Review nit on the staged heap probes: the log line contract says
payload=<N>B (bytes) and inbound-parsed samples bodyBuffer.length
(bytes), but the prepared-stage sampled prepared.body.length — UTF-16
code units for string bodies. Identical for ASCII/base64 payloads,
under-reports multibyte text, and makes the inter-sample payload delta
(unit-mixed) misleading exactly when comparing stages. Buffer.byteLength
is allocation-free, so the debug path stays cheap.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

审核结论:代码可合入;当前 blocked 是仓库级 CI 门禁失配(已立 #3574),与本 PR 无关

做了什么(不止读 diff):rebase 到当前 master b10c525(v0.1.198 release merge,窗口内 handle.ts 零改动,干净)→ 顺手修掉一处评审发现的小缺陷(独立标注 commit 982ecf240)→ 本地 typecheck + 定向套件验证 → 全窗 CI 核查 + 唯一红腿定向归因与重跑。新 head:982ecf240e。

顺手修的小缺陷(commit 982ecf240):prepared 采样点的 payload 字段对 string body 用 .length(UTF-16 字符数),但日志格式契约是 payload=<N>B(字节)、且 inbound-parsed 用的是 Buffer.length(字节)——单位混用会让「相邻采样差值 = 阶段成本」这个核心用法在多字节文本上失真(base64/ASCII 场景无差)。改为 Buffer.byteLength(零分配,debug 路径成本不变)。

结构化八问

  1. 是什么问题:不是 bug 修复,是诊断辅助——0.1.197: A single 25 MiB request drives the heap to the 4 GB ceiling → process aborts → ~10 s per crash-restart cycle (D | 0.1.197:单个 25 MiB 请求把堆吃到 4 GB 上限 → 进程 abort → 约 10 秒一轮崩溃重启(DSH 双窗口同时 Connection error) #3386 的 OOM 根因(图片 base64 三份并存)已钉死,但「每轮到底是哪一段物化出这些字节」进程外不可见,A/B/C 三个修向各自需要这个归因。本 PR 加三个 --debug 门控堆采样点(inbound-parsed / prepared×2 路径)。
  2. 什么原因导致的:不涉及(诊断功能;被诊断的问题见 0.1.197: A single 25 MiB request drives the heap to the 4 GB ceiling → process aborts → ~10 s per crash-restart cycle (D | 0.1.197:单个 25 MiB 请求把堆吃到 4 GB 上限 → 进程 abort → 约 10 秒一轮崩溃重启(DSH 双窗口同时 Connection error) #3386)。
  3. 影响范围与严重程度:off 时零行为变更(memProbe 在碰 process.memoryUsage() 之前早退);on 时每请求多三行 debug 日志。不动 wire、kernel、配置面、缓存。严重程度:无。
  4. 怎么修复的:见上——方向关、rebase、小缺陷修复、本地验证四步。
  5. 是否有回归问题:无。diff 触碰路径的既有行为逐项清点:① debug=false 路径早退、零开销零输出,与既有行为一致;② codex-intercept 分支探针只在 runPrepare() 真正执行时触发(mode≠intercept 或 !gatePre 时 prepared 保持 undefined → 跳过 → 走 normalize-forward 提前返回,根本没跑 prepare)——语义正确,不缺探;③ grep 连续性:site1 用 sessionId、site2/3 用 session.id,核过 getSession/createSession 按传入 id 精确建键、persona fork 的 namespaced key 在 sessionId 派生前就已落在 conversation 上(handle.ts:1004→1258),三点打印同一前缀;④ 时序:site1 位于 content-decoding(handle.ts:368-369)与 toString+JSON.parse(:481-484)之后(:1339),与 PR 描述一致;⑤ log 签名与 handleRequest 参数同型(level: string),走 loggerLog 体系、无 console.error。CI 全窗:新 head 各车道全绿;唯一红腿 = test (windows-latest, 22, 1) 一个 teardown 挂起(lane-decisions-matrix google×dsh-no-tools,TAP fail 0 + cancelled 6 → exit 1)——同测试同 run 的 ubuntu 分片通过、本地 22/22 全绿、我触发的重跑也绿 → CI flake: stream-keepalive tests (#1647) parent-cancelled on loaded Windows shard — no assertion failures #3356 家族 flake(PR [PR] fix: bounded harness close + CI drain-cancel retry shape (#3356) | win22 分片收尾挂起不再无声连坐:harness 有界 close + 重试门识别 drain-cancel 形态 #3563 在途修复,harness 有界 close + drain-cancel 重试形态),与本 PR 零相关(diff 在测试中是空操作)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:无——复用既有 --debug,无 env、无 config 字段,与 PR 描述一致。

合并判定

代码层面 ready:rebase 完成、评审修复落地、本地定向验证全绿(typecheck clean;handle-path 套件 80/80;lane-decisions-matrix 22/22)、CI 重跑后全绿。但 mergeable_state 目前是 blocked,原因不在这个 PR:master 分支保护要求 12 个 test 上下文,而 PR 车道的 core 矩阵只产 7 个(ubuntu 24×(2,3) + windows 24×(1,2,3) 永不产出),拿到精简矩阵的 PR 无论多绿都恒 blocked(#3577、#3565 同样卡住;对照 #3564 为 behind)。这连同另外两个发现(手动专属的 gate-plugins 工作流被每次 push 空触发且必失败;win22 teardown 挂起 flake)一并立在 #3574(查重修订:A 项与在途 PR #3563 重复,已降级为对它的验证项)。

等你把保护规则对齐 core 矩阵契约(或给该 PR 打 ci:full),即可人工合并:#3552 。合入后的用法:#3386 复现机器开一次 --debug,grep mem-probe 按 sessionId 拉出三段采样,差值即各阶段成本——A(每轮图片硬上限)与 C(base64/data-URI 去重)的取舍就有实测依据了。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4525(master 6d7c587,合并方式 merge commit,提交保留原作者署名)。本 PR 关单收尾;若列车中对应提交出问题,回滚点为 #4525 单提交 revert。

@ranxianglei ranxianglei changed the title debug: staged heap probes over the request processing window (--debug gated, zero behavior change) [PR] debug: staged heap probes over the request processing window (--debug gated, zero behavior change) Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant