Skip to content

[PR] fix: decode zstd requests asynchronously when supported - #6489

Merged
ranxianglei merged 18 commits into
ranxianglei:masterfrom
Afloat16:2026-11-12_async-zstd-decode
Nov 14, 2026
Merged

ranxianglei merged 18 commits into
ranxianglei:masterfrom
Afloat16:2026-11-12_async-zstd-decode

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #6488

Problem

zstd request decoding uses a synchronous fzstd push on every supported runtime. Although the helper returns a promise, the entire compressed body is processed before the call returns. A client cancellation immediately after starting the decode is therefore ignored, and large requests occupy the server event loop throughout decompression.

Change

Use Node's native createZstdDecompress stream when available, through the same streamDecode helper used by gzip, Brotli and deflate. Validate complete Zstd frame boundaries and decode each frame asynchronously so Node 22 also preserves concatenated input and rejects truncation. This retains the existing decoded-byte limit, cumulative admission observer and abort cleanup. Keep the fzstd path when the native factory is absent, including supported older Node runtimes.

No new dependency or configuration setting is introduced. Corrupt native input remains a decode failure; it is not retried through another decoder.

Initial validation (Node 24.22.1)

  • Base: 9674cfa. The two new cancellation cases fail on unchanged master; 13 other tests pass in that baseline run.
  • Initial Node 24.22.1 validation: 27 tests passed across content-encoding-zstd, content-encoding, request-body-budget, decode-fail-passthrough, issue4829-zstd-fork-gate and issue5794-ingest-cap. These include real local HTTP requests, hard size limits, admission-error identity, concurrent cancellation, truncated input, stacked codings and concatenated frames.
  • A child process with the native factory removed verifies the older-runtime fallback, caps, admission errors and pre-aborted requests. This simulates feature absence on Node 24.22.1; an actual Node 20 binary was not run.
  • npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph and git diff --check pass.
  • Initial Node 24.22.1 benchmark (before the compatibility follow-up): the committed script can be run with node --import tsx scripts/bench-zstd-decode.mjs. On Linux x64 / Node 24.22.1, a repeated 16 MiB sample (1,609 compressed bytes; one warm-up, five measured calls) changed median synchronous call time from 63.28 ms to 0.17 ms and total decode time from 63.31 ms to 27.11 ms. These are synthetic, machine-dependent measurements, not a CI timing threshold.
  • Zero-token Codex preflight found the CLI and built artifact, but the local upstream /models probe was unreachable. The real-model Codex suite was skipped; upstream CI had not been verified at that stage. On older runtimes without native zstd, the retained fzstd path remains synchronous.

Prior work and tracking

Checked all 61 Afloat16 PRs, all 159 fork branch names, the content-encoding commit history and upstream searches for zstd/native/fzstd/asynchronous decoding. #4832 adds content-encoding support to the fork-adoption gate; it does not replace the synchronous zstd decoder. No equivalent patch was found.

CI follow-up

Registered tests/content-encoding-zstd.test.ts in LIVE_FILES in 829a7853, then merged current master (fcc2e848) in 50926b86. The registry conflict was resolved by retaining all upstream entries at their existing zone slots and appending the zstd case. At that integration, the decoder, benchmark and regression-test files were unchanged from the previous PR head.

The fallback case changes the zstd factory only inside its isolated Node child and binds no sockets. On the merged tree (Linux / Node 24.22.1), the actual CI runner passes its global process-spawn registration check and all 7 tests in the selected zstd file. Full typecheck, production build and git diff --check also pass.

CI run 43367878116 on 50926b86 completed with the Node 22 failures addressed below.

Node 22 compatibility follow-up

Commit 7341f7d1 fixes the two concrete failures from Ubuntu / Node 22 shard 2 and Windows / Node 22 shard 4. Both runners used Node 22.23.4: concatenated frames returned only the first 33 bytes instead of 66, and truncated input incorrectly succeeded. The preceding Node 24 lanes passed.

This matches the native decoder issues tracked in nodejs/node#64741 and nodejs/node#64592. Explicit finishFlush: ZSTD_e_end does not fix the Node 22 behavior. The follow-up checks frame extents according to RFC 8878 §3.1, then sends complete frames through the existing native asynchronous decoder one at a time.

The boundary walk handles variable frame headers, raw/RLE/compressed blocks, optional checksums, and skippable frames. Native zstd continues to validate compressed contents and checksums. The walk does not decompress blocks, and the next frame starts only after the preceding decode finishes. Decoded-byte accounting is cumulative across frames; overflow still reports the original configured limit, and cancellation/admission errors retain their identity. No dependency was added.

Verification of the follow-up

  • Reproduced the original two failures locally with the exact Node 22.23.4 binary: the original seven-case suite passed 5/7.
  • With the expanded 17-case regression suite and unchanged production code, Node 22.23.4 passed 6/17 while Node 24.22.1 passed 17/17. The failed Node 22 cases cover concatenation, incomplete later frames, truncated headers/payloads/checksums, skippable data, and a corrupt later checksum.
  • With the fix, 17/17 Zstd cases pass on both Node 22.23.4 and Node 24.22.1.
  • The same six related suites listed above were then run after dependency builds had completed, separately on each runtime: 37/37 pass on Node 22.23.4 and 37/37 pass on Node 24.22.1, including real local HTTP checks.
  • npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph, and git diff --check pass.
  • An independent format comparison also checked eight valid frame-header variants on each runtime, including the content-size and dictionary-ID field widths; all produced the same bytes as the native reference decoder.
  • The existing 16 MiB benchmark on Node 22.23.4, with one warm-up and five measured calls, reported a 0.57 ms median synchronous call time and 137.08 ms median total decode time. These are synthetic measurements from this environment, not CI timing requirements.

The published tree is 929bfe6dca13059e1904df2438c6b86fc53b3708, matching the tested and independently reviewed candidate. The commit is a normal child of 50926b86, which already contains upstream master fcc2e848.

Upstream validation of the final head

Upstream CI run 43375699576 completed successfully on the current PR head 7341f7d1 with the default CORE matrix.

The Windows / Node 22 shard 3 log records all 17/17 Zstd cases passing with no skips, including the native cancellation cases and the concatenation/truncation regressions. The full shard reports 2,643 passed, zero failed, and six unrelated tests skipped. It ran on Windows Server 2025 with Node 22.23.4 and npm 10.9.9. The file moved to shard 3 under the current workload allocation; the previously failing shard 4 also passed.

GitHub checked out test-merge commit 2f4f93efa760ae7e51e717f0b9b2fec8d39fb780, whose parents are base fcc2e848 and PR head 7341f7d1. Its tree is exactly 929bfe6dca13059e1904df2438c6b86fc53b3708, the same tree tested locally and published above.

Review status

Ready for maintainer review. Current head is 7341f7d1; refreshed hosted results are available in the Checks tab. The upstream CORE matrix on this head has passed; real-model E2E remains unverified.

Please apply the existing ci:full label to run the FULL matrix, including the required Windows / Node 24 checks. The default CORE matrix does not produce those checks.

@ranxianglei ranxianglei changed the title fix: decode zstd requests asynchronously when supported [PR] fix: decode zstd requests asynchronously when supported Nov 13, 2026
Register the process-spawning zstd regression in LIVE_FILES so the CI runner can start the test matrix. Append the entry to preserve existing zone slots; the child changes only its own zlib exports and binds no sockets.

The actual CI runner's global registration check and all 7 zstd cases pass locally. Typecheck, build and diff checks also pass.
Retain the upstream pack-install and CI-janitor entries at their existing slots, then append the zstd fallback regression. The original decoder, benchmark and regression test blobs are unchanged.

On the merged tree, the actual CI runner passes its global registration check and all 7 zstd cases. Full typecheck, production build and net diff checks pass.
@Afloat16
Afloat16 marked this pull request as ready for review November 13, 2026 04:57
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复. The only zstd-related items in the repo are #6488 (the source issue) and this PR. No prior work on asynchronous zstd decoding exists (searched open + closed issues/PRs; the author also checked all 61 Afloat16 PRs + 159 fork branches + the content-encoding commit history).

Live state (verified 2026-11-13, ~07:05 UTC): head 7341f7d1 (4 commits, 4 files, +264/−5), mergeable: true (no conflict), not a draft, CI fully green — all 11 test lanes (ubuntu 22/24 ×3, windows 22 ×5) + static-gates / docs-sync / mysql-contract / test-import-graph / unused-exports success. The 12 skipped lanes are the e2e/registry lanes that skip by design on fork PRs (no secret access) and on files outside the e2e trigger list (src/content-encoding.ts is not a pipeline hot file per reference/testing.md).

① Direction. Issue #6488 is real and at root cause: decodeZstd pushed the whole body through fzstd's synchronous API, so the returned promise resolved only after full decompression — client cancellation during decode was ignored and the event loop was blocked for the whole body. The fix (native createZstdDecompress stream through the existing streamDecode helper, src/content-encoding.ts:125-143) removes the producer of the defect, not a mask. Verified locally (Node 22.23.2): the two cancellation tests fail on unchanged master and pass after the fix — genuine regression tests.

② Regression review. The original PR head (50926b86) had a real Node 22 regression: it passed the whole buffer to one native stream, but Node 22's native zstd decoder (a) drops frames after the first in a multi-frame buffer and (b) silently accepts truncated frames — both confirmed by local probes, and both are known upstream bugs (nodejs/node#64741 open, #64592 closed — verified real). The CI node-22 lanes caught it; the follow-up commit 7341f7d1 ("fix: preserve complete zstd frames on Node 22") adds an RFC 8878 frame-extent parser (zstdFrames, src/content-encoding.ts:71-123) that validates every frame's extent without decompressing, then decodes complete frames separately under a cumulative byte cap. I verified the parser bit-for-bit against RFC 8878 (Table 3 descriptor bits, Table 4 FCS sizes, Table 5 DID sizes, window-descriptor presence, Table 9 block header, RLE = 1-byte content, skippable-frame range 0x184D2A50–0x184D2A5F) — the layout is correct. All 17 tests pass locally on Node 22.23.2; the 5 related existing suites (content-encoding, request-body-budget, decode-fail-passthrough, issue4829-zstd-fork-gate, issue5794-ingest-cap) pass 20/20 against the head code.

Established behavior audited and preserved:

  • fzstd fallback path (runtimes without native zstd, Node < 22.15): byte-identical to master.
  • Decode-fail → passthrough: corrupt input now yields parser errors ("truncated zstd frame" / "invalid zstd frame magic" / "reserved zstd frame bit" / "reserved zstd block type") or native ZSTD_error_* instead of fzstd errors — all plain Errors, and the caller (src/server/handle.ts:517) treats every non-special error as decode-fail → forward the original encoded bytes verbatim (Follow-up from #617: robust session identity + no-spurious-400 for header-less clients (e.g. ZCode) #619). Behavior unchanged.
  • Byte cap: per-frame max in streamDecode + cumulative cap in the onBytes wrapper (src/content-encoding.ts:133-137) — pinned by a cross-frame cap test.
  • Admission observer: receives the cumulative total, monotonic — pinned by test.
  • Abort: pre-aborted (entry throwIfAborted + streamDecode pre-check), mid-decode (per-frame abort listener), between frames (per-frame throwIfAborted) — all pinned by tests.

Known limitation (not a blocker): on Node 22 only, the parser trusts block sizes for frame extent; a multi-frame input with a corrupted block size that coincidentally lands exactly on the next frame's magic could be partially decoded (Node 22's native decoder does not detect the truncation). Single-frame input — the common case — always fails closed (leftover bytes → magic check throws). Requires adversarially precise corruption; not practically reachable.

③ Code. The namespace import * as zlib (src/content-encoding.ts:1) is required — a named import of createZstdDecompress would throw at ESM link time on older Node. Correct. The parser is clean, well-commented (cites the RFC + the upstream Node issues), and matches the file's existing style. The scripts/live-files.mjs registration is correct (the SPAWN_RE tripwire in run-ci-tests.mjs requires process-spawning test files to be listed). The benchmark script is manual-only, not a CI threshold — fine.

④ Tests. Real tests with teeth: the two cancellation cases are genuine regression tests (fail on master); both gate directions are asserted (triggers: cap/abort/admission across frames; does-not-trigger: unknown-size, checksummed, interleaved skippable, RLE, and compressed frames all decode correctly); 7 truncation rejections pin the fail-closed behavior the Node 22 native decoder alone would miss; the child-process case (native factory removed via syncBuiltinESMExports) verifies the fzstd fallback. The fix commit's 59 new test lines pin the exact semantics: cumulative onBytes (monotonic + final value), cross-frame cap, admission-error identity in the second frame, abort reason in the second frame, and per-frame checksum validation.

⑤ Diff. 4 files, +264/−5, every line related to the PR's purpose. No unrelated changes, no whitespace churn, no lock-file churn. Clean.

Behavior changes (flagged per policy). On native-capable runtimes (Node ≥ 22.15.0 / ≥ 23.8.0):

  1. zstd request decoding is now asynchronous — the event loop is no longer blocked during decompression (the intended fix; author's benchmark: 16 MiB median sync call 63.28 ms → 0.17 ms).
  2. Client cancellation during decode now takes effect (previously ignored until full decompression).
  3. Corrupt-zstd error type changed from an fzstd error to a parser/zlib error (caller behavior unchanged — passthrough).
    No new dependency, no new config, no new env var.

Auto-merge gate (§7.4): not an auto-merge candidate — behavior change (async decode, new error types) plus a hand-written binary parser; the human review tier is appropriate. Otherwise mergeable: single-module scope, regression tests in place, CI green on the rebased head, no config/persistence/wire-protocol change.

Verdict: mergeable (human merge; mergeable_state: blocked = branch-protection review, not a conflict). The CI engineer's fix commit resolved the one real regression in the original PR with comprehensive tests.


结构化八问

  1. 是什么问题: 在原生支持 zstd 的运行时(Node ≥ 22.15)上,请求体 zstd 解码走 fzstd 同步 push——promise 虽然返回了,但整个压缩体在返回前已解压完,客户端解码中途取消被忽略,大请求全程占住服务端事件循环。
  2. 什么原因导致的: 解码器选型——fzstd 只暴露同步 API,decodeZstd 同步 push 完整个 body 才 resolve;取消信号与字节回调都挂在同步循环里,无法在解码中途生效。
  3. 影响范围与严重程度: 所有原生 capable 运行时上带 content-encoding: zstd 的请求;严重程度:特定条件下有问题——大 zstd 请求期间事件循环被阻塞(其他请求延迟)、取消无效;不是不可用,但大 body 场景体验受损。
  4. 怎么修复的: 改用 Node 原生 createZstdDecompress 流,走既有 streamDecode 助手(与 gzip/br/deflate 同一条路),保留字节上限、admission 观察器、abort 清理;无原生工厂的老运行时保留原 fzstd 路径(逐字节未动)。初版在 Node 22 上有真回归(原生解码器丢多帧、静默接受截断帧,zstd decoding complete frame in read stream halts stream nodejs/node#64741/#64592),CI node-22 车道抓出后,补了一个 RFC 8878 帧边界解析器先校验每帧范围再逐帧解码,累计上限在 onBytes 包装里强制。
  5. 是否有回归问题: 审过的既有行为——fzstd 回退路径(逐字节未动)、解码失败→passthrough 语义(handle.ts:517 对非特殊错误一律转发原始编码字节,Follow-up from #617: robust session identity + no-spurious-400 for header-less clients (e.g. ZCode) #619)、字节上限/admission/abort 语义(测试钉死,含跨帧累计)——全部保持。初版 PR 的 Node 22 回归已被 CI 抓出并修复(17 个测试本地 Node 22.23.2 全过,5 个相关既有套件 20/20)。残留理论缺口:仅 Node 22 + 多帧 + 对抗性精确损坏(块大小恰好落在下一帧 magic 上)可能部分解码;单帧输入恒 fail-closed。
  6. 是否破坏缓存: 不涉及——请求体解码在会话/缓存层之下,不触碰 prompt cache、重试、增量游标、去重键。
  7. 是否改了设计方向: 不涉及架构转向;只是把 zstd 从「同步特例」拉齐到「与 gzip/br/deflate 同一条流式解码路径」,方向与既有设计一致。
  8. 是否引入了新的配置: 无新配置、无新依赖、无新 env。

收工判定: 可合(人工合入)。CI 全绿、无冲突、回归测试真实有效;唯一真回归(Node 22 多帧/截断)已由 CI 工程师的修复 commit 收敛。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 已全绿。本轮 CI 工程处置摘要(head 7341f7d1,镜像缓存 07:14 UTC 核实:30 项检查 0 failed / 0 pending):

  1. 是什么问题:PR 的 11 个 test 分片(ubuntu/windows × node 22/24)全部红,测试根本没跑起来。
  2. 什么原因导致的:新增的 tests/content-encoding-zstd.test.ts 会 spawn 子进程,但未在 scripts/live-files.mjs 的 LIVE_FILES 登记,CI 的 process-spawning 注册检查(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248)在每个分片开跑前直接报错退出。已拉原始红 job 日志确认,11 个分片错误签名完全一致。
  3. 影响范围与严重程度:仅本 PR 的 CI 门禁,不涉及已发布行为;修复后全绿。
  4. 怎么修复的:作者在本轮期间自行推了 3 个 commit,我逐一核验:① 829a7853 把测试文件登记进 LIVE_FILES(治本,注册检查通过);② 50926b86 合并 master(分支落后,保留双方登记项);③ 7341f7d1 修复 Node 22 原生 zstd 路径的两个真实行为缺陷——原生 createZstdDecompress 会静默丢弃同一 buffer 里的后续帧、并接受截断帧不报错(zstd decoding complete frame in read stream halts stream nodejs/node#64741、#64592),改为按 RFC 8878 先扫描并校验帧边界、再逐帧解码,另加 10 个回归用例(未知长度帧、校验和帧、skippable 帧、RLE/压缩块、各类截断)。我本地(Node 22.23.2)独立复现过这两个缺陷后才看到该 commit,方向一致。
  5. 是否有回归问题:本地定向验证——17 个 zstd 用例全过,相邻 content-encoding.test.ts + request-body-budget.test.ts 8 例全过;CI 30 项检查全绿(含 static-gates、lane-guard、architecture-docs 等)。fzstd 回退路径行为未动。
  6. 是否破坏缓存:不涉及——纯请求体解码路径,与缓存/重试/增量游标无关。
  7. 是否改了设计方向:不涉及——方向仍是「原生 zstd 可用时走异步原生解码、否则回退 fzstd」,第 3 个 commit 是对该方向的补全(帧边界校验),不是转向。
  8. 是否引入了新的配置:不涉及。

备注:mergeable: true(无冲突),mergeable_state: blocked 是等人工 review/批准,非 CI 问题。本轮我未推任何 commit(修复均由作者完成,我做的是核验 + 本地验证)。结论:CI 侧收工,可进入人工 review。

ework-agent and others added 14 commits November 14, 2026 04:35
The suite baseline preload (preload-storage-baseline.mjs) pins
BILI_STORAGE_UNIFIED="legacy" for the broad corpus. After ranxianglei#6480 flipped the
storage default to pure-sqlite, the notice-registry suite's visibleNotices
assertions expect the pure-sqlite default notice, which is gated on
unifiedSelectionIsSqlite (unified "sqlite" or absent) — the preload's
"legacy" made the notice silent, so the suite was born-red on master. Pin
the selection explicitly (ranxianglei#4011 lesson: a test that depends on the production
default passes locally but fails in CI).
…test

The dsh plugin lane (ci-e2e-dsh.yml) runs node --import tsx --test
WITHOUT the preload-storage-baseline.mjs preload, so BILI_STORAGE_UNIFIED
is unset and the storage mode defaults to 'full'. Under the pure-sqlite
default flip (5a80e2b, ranxianglei#6480), 'full' mode turns the JSON envelope
writer OFF — so the .content-store.json sidecar is never written, and
the test's readdirSync(namespace) hits ENOENT.

Pin BILI_STORAGE_JSON_WRITER=on in the test so the content-store
sidecar is written regardless of the storage mode. This keeps the
corrupted-store detection path exercised in both full and legacy modes.

Closes: dsh plugin lane red on master (run 43532980164, job 132255459726)
The pure-sqlite default flip (5a80e2b, ranxianglei#6480) turns the JSON envelope
writer OFF, so the .json session file this lane asserts on is never
written and 'proxy must persist the session' / 'no persisted session
carried compressed blocks' fail after the 20s wait. Pin the writer ON in
the spawned client's env (the bootstrapped proxy inherits it) so the
envelope this test was written for is still produced.

Closes: native opencode lane red on master (run 43547593803, job 132302438610)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode lane:
the JSON envelope writer is OFF, so 'proxy must persist the session
within 20000ms' fails — the on-disk .json session file is never written.
Pin the writer ON in the spawned dsh client's env (the bootstrapped
proxy inherits it).

Closes: native dsh lane red on master (run 43547497015, job 132302138843)
…e2e lane

Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode/dsh
lanes: the JSON envelope writer is OFF, so 'session persisted' fails —
the on-disk .json session file is never written. Pin the writer ON in
the spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi codex-responses WS lane red on master (run 43547626291, job 132302540960)
…lane

4th lane of the pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the
JSON envelope writer is OFF, so 'proxy must persist the session' and
'at least one persisted session must carry compressed blocks' fail — the
on-disk .json session file is never written. Pin the writer ON in the
spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi package-native lane red on ci-fixes (run 43558179054, job 132336647955)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON writer
is OFF, so the per-session CCR content-store sidecar (.content-store.json)
that 'child persists its own companion and survives a proxy restart'
asserts on is never written (0 !== 1). The earlier b1bc17b pin only
covered tests/plugin-fork.test.ts, not this file. Pin the writer ON at
module load (before any SessionStore is constructed).

Closes: dsh plugin lane red on ci-fixes (run 43558179102, job 132336519539)
…lane

Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so 'proxy must persist the session' fails — the on-disk
.json session file is never written. Pin the writer ON in the spawned
opencode client's env (the bootstrapped proxy inherits it).
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so the on-disk .json session envelope this lane asserts on
(exactly ONE persisted session under the raw key) is never written. Pin
the writer ON in the spawned dsh client's env.
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so the on-disk .json session file this lane asserts on
(state survival across the OLD→NEW update) is never written. Pin the
writer ON in the spawned proxy's env (both update phases share envBase).
Resolve CI baseline conflicts using the current upstream test setup.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review verdict: APPROVE — PR #6489 "fix: decode zstd requests asynchronously when supported" (head 1cd78a38f).

I did an independent end-to-end review — re-derived every conclusion from the diff plus runtime evidence on Node 22.23.2, not from the prior review comments. The fix is correct, the tests are genuine regression tests, the diff is clean, and house conventions are followed. No kernel/ touch, so no authorization gate.

What I verified (independent):

  1. Frame-parser bit layout is correct. I flipped each zstd frame-descriptor bit against the native decoder and confirmed the parser's interpretation matches libzstd exactly: dictionaryFlag = descriptor & 3 (bits 0-1 → "Dictionary mismatch"), checksum = descriptor & 4 (bit 2, accepted), reserved = descriptor & 8 (bit 3 → "Unsupported frame parameter"), singleSegment = descriptor & 0x20 (bit 5), FCS size = descriptor >>> 6 (bits 6-7). Bit 4 (0x10) is accepted by libzstd (lenient) and is correctly left unchecked — making the reserved check stricter would reject valid frames (a regression), so descriptor & 8 is correct as-is.
  2. The native decoder bugs are real. On Node 22.23.2: (a) a 2-frame input of small (33-byte) frames decodes to 33 bytes instead of 66 (second frame silently dropped); (b) a truncated frame is partially decoded (4063232 bytes out) instead of rejected. Both are fixed by the parser's frame-splitting + extent validation.
  3. The parser is correct. 30/30 round-trip + exhaustive-truncation across 15 frame variants (sizes 0–200000, RLE, random high-entropy, multi-frame concat, skippable-frame interleaving). No gaps.
  4. Caller behavior preserved. src/server/handle.ts:516-525 rethrows only DecompressedTooLargeError / IngestBodyTooLargeError / DecodedRequestBusyError / abort; every other decode error (the parser's new Errors and native zlib errors included) falls to protocol = null (verbatim passthrough) — the SAME treatment as fzstd errors on master. The 413 / 503 / cancellation paths are unchanged.
  5. fzstd fallback byte-identical. The else branch (no native factory) is unchanged from master.
  6. The 2 cancellation tests are genuine regression tests. I ran them against master's fzstd sync content-encoding.ts and confirmed they FAIL (not ok 1, not ok 2); they pass with the fix.
  7. All 17 PR tests pass on Node 22.23.2.
  8. Diff is clean. The PR's real contribution is exactly 4 files (src/content-encoding.ts, tests/content-encoding-zstd.test.ts new, scripts/bench-zstd-decode.mjs new, scripts/live-files.mjs +1 entry); the large merged-in diff is upstream master, which does NOT touch any of the 4 PR files. The live-files.mjs registration is correct (the test spawns a Node child).

Behavior changes (disclosed in the PR body, confirmed): zstd decode is now async on native runtimes (event loop unblocked; author bench 16 MiB median 63.28 ms → 0.17 ms); cancellation now works (was a no-op on the sync fzstd path); corrupt-zstd error type changed fzstd → parser/zlib (caller behavior unchanged = passthrough). No new dependency, config, or env.

No small findings to fix — bit layout, reserved check, byte-cap logic, and test registration are all correct as-is.

Context note (not a blocker): the owner folded #6488 into aggregation issue #6449 (root-cause performance-memory). That is a coordination decision for the owner; this review is on the PR's technical merit, which is clean.


8问(结构化审核):

  1. 是什么问题:在支持原生 zstd 的 Node 运行时上,zstd 请求体解码走 fzstd 同步路径,阻塞事件循环且无法响应取消信号。
  2. 什么原因导致的:src/content-encoding.ts 的 decodeZstd 无条件使用 fzstd 同步 push(input, true),不利用 Node 22+ 原生 zlib.createZstdDecompress 异步流式解码器;同步路径在函数返回前处理完整个 body,abort 无法介入。
  3. 影响范围与严重程度:所有带 content-encoding: zstd 的请求(原生 zstd 可用的运行时)。性能/体验问题(事件循环阻塞、取消无效),非数据损坏、非不可用。
  4. 怎么修复的:检测到原生工厂时,先用 RFC 8878 §3.1 帧边界解析器(只验证 extent、不解压)拆帧,再逐帧用原生异步解码器解码,跨帧累计字节上限;无原生工厂时回退到与 master 逐字节一致的 fzstd 同步路径。
  5. 是否有回归问题:审过的既有行为——gzip/br/deflate(命名空间访问,行为不变)、fzstd 回退(逐字节一致)、字节上限/准入/中止(保留,测试钉住)、堆叠编码(保留,测试钉住)、调用方错误处理(passthrough 语义不变)——均无回归。已知理论残留(非阻塞、不可实际触达、仅 Node 22):解析器信任块大小算 extent,若多帧输入中某块大小被对抗性损坏且恰好落在下一帧 magic 上,Node 22 宽松截断接受可能部分解码;Node 24 解码器 fail-closed,不受影响。
  6. 是否破坏缓存:不涉及(无缓存/重试/游标/去重键变更)。
  7. 是否改了设计方向:不涉及(解码器内部实现变更,架构/边界/数据流不变)。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值;无新依赖——fzstd 仍是 devDependency,zstd 仍是可选编码)。

结论:APPROVE — 代码正确、测试真实、diff 干净、符合仓库规范,无 kernel/ 触碰,无授权门。合并判定:可合(owner 自行决定直接合 #6489 还是并入 #6449 聚合)。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE. Independent end-to-end review: the RFC 8878 frame parser's bit layout matches libzstd exactly (verified by flipping each descriptor bit against the native decoder on Node 22.23.2); the native decoder bugs it works around (small-frame multi-frame drop, truncated-frame acceptance) are real; the parser passed 30/30 round-trip + exhaustive-truncation across 15 frame variants; the caller's passthrough error semantics (handle.ts:516-525) are preserved; the fzstd fallback is byte-identical; the 2 cancellation tests are genuine regression tests (fail on master, pass with the fix); all 17 PR tests pass; the diff is clean (4 files, the merged-in upstream master touches none of them). No kernel/ touch, no new dep/config/env. Full 8-question review in the issue thread.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE (re-review, new head 889a403). The PR's code is byte-identical to the previously-reviewed head 1cd78a3: the new head is a merge of master (release v0.1.214, gate-recorder pipe-leak fix #6671, storage/e2e updates) that touches NONE of the PR's 4 files. Verified src/content-encoding.ts, tests/content-encoding-zstd.test.ts, scripts/bench-zstd-decode.mjs byte-identical; the live-files.mjs PR entry (content-encoding-zstd) is unchanged (the only live-files delta vs the old head is the storage-open-busy-lock entry, already on origin/master). All prior findings carry over: RFC 8878 frame-parser bit layout matches libzstd (empirically verified on Node 22), the native decoder bugs it works around are real, parser passed 30/30 round-trip + exhaustive-truncation, caller passthrough semantics preserved, fzstd fallback byte-identical, the 2 cancellation tests are genuine regression tests, all 17 PR tests pass, clean 4-file diff. No kernel/ touch. CI: 14 test shards queued on the new head (unchanged code + green master base -> expected green).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Re-review of new head 889a40305: APPROVE — code is byte-identical to the previously-reviewed head 1cd78a38f.

The new head is a merge of master (release v0.1.214, gate-recorder pipe-leak fix #6671, storage/e2e updates) that touches NONE of the PR's 4 files. Verified:

  • src/content-encoding.ts, tests/content-encoding-zstd.test.ts, scripts/bench-zstd-decode.mjs — byte-identical to the old head.
  • scripts/live-files.mjs — the PR's own entry (content-encoding-zstd) is unchanged; the only delta vs the old head is the storage-open-busy-lock entry, which is already on origin/master (from the merged-in release).

So the full PR contribution vs origin/master is still exactly 4 files (+264/−5), and every finding from my prior review (floor 15) carries over in full: RFC 8878 frame-parser bit layout matches libzstd (empirically verified by flipping each descriptor bit on Node 22.23.2), the native decoder bugs it works around (small-frame multi-frame drop, truncated-frame acceptance) are real, the parser passed 30/30 round-trip + exhaustive-truncation across 15 frame variants, the caller's passthrough error semantics (src/server/handle.ts:516-525) are preserved, the fzstd fallback is byte-identical, the 2 cancellation tests are genuine regression tests (fail on master, pass with the fix), all 17 PR tests pass, and the diff is clean. No kernel/ touch, so no authorization gate.

CI status (new head): 14 test shards (ubuntu 22/24 ×3, windows 22 ×5) + static-gates ×2 + mysql-contract are still queued on the new head; 12 e2e/registry lanes skip by design on fork PRs, 4 static checks already green. Given the unchanged code + green master base, these are expected to land green.


8问(re-review — 代码未变,结论同 floor 15):

  1. 是什么问题:原生 zstd 运行时上 zstd 请求体解码走 fzstd 同步路径,阻塞事件循环且无法响应取消信号。
  2. 什么原因导致的:decodeZstd 无条件用 fzstd 同步 push(input, true),不利用 Node 22+ 原生异步 createZstdDecompress;同步路径在返回前处理完整个 body,abort 无法介入。
  3. 影响范围与严重程度:所有带 content-encoding: zstd 的请求(原生 zstd 可用运行时)。性能/体验问题(事件循环阻塞、取消无效),非数据损坏、非不可用。
  4. 怎么修复的:原生工厂存在时先用 RFC 8878 §3.1 帧边界解析器(只验证 extent、不解压)拆帧,再逐帧用原生异步解码器解码,跨帧累计字节上限;无原生工厂时回退到与 master 逐字节一致的 fzstd 同步路径。
  5. 是否有回归问题:无。gzip/br/deflate(命名空间访问,行为不变)、fzstd 回退(逐字节一致)、字节上限/准入/中止(保留,测试钉住)、堆叠编码(保留,测试钉住)、调用方 passthrough 语义(handle.ts:516-525 不变)——均保持(floor 15 逐项验证)。已知理论残留(非阻塞、仅 Node 22、对抗性精确损坏才可触达、Node 24 fail-closed)同 floor 15。
  6. 是否破坏缓存:不涉及(无缓存/重试/游标/去重键变更)。
  7. 是否改了设计方向:不涉及(解码器内部实现变更,架构/边界/数据流不变)。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值;无新依赖)。

结论:APPROVE — 代码与已审 head 逐字节一致、无 kernel/ 触碰、无授权门、无小发现需修。合并判定:可合(owner 决定直接合 #6489 还是并入 #6449 聚合)。

@ranxianglei
ranxianglei merged commit ac2eb2d into ranxianglei:master Nov 14, 2026
50 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit ac2eb2db3c3 (Merge Afloat16/2026-11-12_async-zstd-decode (PR #6489)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6488 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zstd request decoding blocks cancellation on native-capable Node runtimes

2 participants