Skip to content

[PR] fix(web): enforce decoded blog download byte limits - #7341

Merged
ranxianglei merged 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-20_cap-blog-download-streams
Nov 20, 2026
Merged

ranxianglei merged 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-20_cap-blog-download-streams

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #7339

Problem

Blog refresh already has 512 KiB text and 2 MiB image limits, but raw downloads apply them after buffering the complete response. Missing or understated Content-Length therefore allows oversized downloads to run to EOF before being rejected. Early HTTP/status/declared-length exits also leave the response body unread.

The contents-API fallback has the opposite boundary error: its encoded-length estimate counts base64 padding and line breaks as file bytes, so valid 524287-byte and 524288-byte documents are rejected.

Changes

  • Read raw response chunks against the existing byte cap, cancel on overflow, and release the reader.
  • Cancel unread bodies before stopping their idle watchdog.
  • Measure whitespace-normalized base64's decoded length before allocation, retaining the post-decode guard.
  • Preserve text-source fallback, clean image misses, exact-limit acceptance, and the last good overlay.

All changes are confined to the blog-refresh module and its regression suite.

Validation

Tested base: 6453feeeb842bd1e471d875bf00287e5036a96c7 (billion-context 0.1.216).

  • Stream regression before fix: 4 failed, 1 compatibility case passed.
  • Base64 boundary regression before fix: 4 failed, 2 oversized-input cases passed.
  • After fix: node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx --test tests/web-blog-refresh.test.ts — 38 passed.
  • npm run build — passed.
  • npm run typecheck — passed.
  • npm run check:unused-exports — passed (existing waivers retained).
  • npm run check:test-import-graph — passed.
  • git diff --check — passed.

The full-suite verdict remains the repository's remote CI on the submitted head.

Scope

No configuration, persistence-format, wire-protocol, kernel, or model-facing-text change. The limit values and source ordering are unchanged. This bounds raw file streaming; it does not introduce a new cap on the contents-API JSON envelope.

Prior work checked: #4972 (Content-Length prechecks) and #5394 (source fallback), plus open/closed issue and PR searches for streamed blog downloads and base64 size boundaries.

Bound raw document and image reads before buffering an oversized response,
cancel rejected bodies, and measure contents-API base64 payloads after
removing whitespace and accounting for padding. Preserve fallback and
last-good-overlay semantics at the existing byte limits.
@ranxianglei ranxianglei changed the title fix(web): enforce decoded blog download byte limits [PR] fix(web): enforce decoded blog download byte limits Nov 20, 2026
@ranxianglei ranxianglei added bug Something isn't working p2 scope/ui labels Nov 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Dedup screening: not a duplicate — no parallel open PR targets this thread; the PR correctly references source issue #7339 and states it checked prior related work (#4972 Content-Length prechecks, #5394 source fallback) without duplicating it.
Labels: bug + p2 + scope/ui (kept existing root-cause) — a real defect but on a fail-soft, non-critical web-panel path; not a usability blocker and not a cache leak, so p2 not p1.

Review verdict: mergeable

I independently reviewed PR head 8a29d3a (base 6453fee, billion-context 0.1.216) — I did not copy the issue-side conclusions.

① Direction (source issue #7339) — both defects are real:

  • I reproduced the base64 boundary bug locally with node: for 524287/524288-byte docs the old estimate Math.ceil(699052*3/4)=524289 → the old code wrongly rejected both valid docs; Buffer.byteLength(compact,"base64") matches the actual decoded length exactly (delta −1/0/+1, one each). The pathological "A===" only overestimates (byteLength=1 vs actual 0) — safe, and GitHub never emits it.
  • Streamed cap: the old Buffer.from(await arrayBuffer()) buffered to EOF before checking, so a missing/understated Content-Length let an oversized download run to EOF before rejection — exactly [Bug] Blog refresh byte caps over-buffer streams and reject valid base64 documents #7339's memory vector.

② Regression (existing-behavior inventory) — no regressions:

  • Valid content is byte-identical; oversized content is rejected earlier (intended, disclosed); boundary docs 524287/524288 are now accepted (intended fix, disclosed).
  • Cap values (512 KiB text / 2 MiB image / 8 MiB total budget), source ordering, and fail-soft semantics are unchanged.
  • No config / persistence / wire-protocol / kernel / model-facing-text change — the PR's scope statement is accurate.
  • High-risk patterns walked: no semantic drift, no fail-fast→swallow, no threshold/timing/output-format drift.

③ Code — high overall quality (DRY helpers, no dead code, no magic numbers, tests with real assertions):

④ Test validity — 38/38 pass (I re-ran on the PR head locally). The new cases have teeth: 6 base64-boundary (delta −1/0/+1 × wrapped/plain), 2 streamed-text-cap, oversized-image-cancel, exact-cap-retained, content-length-cancel. Not tautologies — they'd catch a regression.

⑤ One ⑤b code-cleanliness finding (external submissions are strict): the line if (!result.response.bodyUsed) await result.response.body?.cancel().catch(() => {}); is duplicated verbatim across all 5 finally blocks (≥3 places must extract a shared helper).

  • I extracted it into a helper releaseUnreadBody(response) locally and verified it: 38/38 tests pass + typecheck clean, commit 79807cedbef (behavior-preserving).
  • But the push to the PR branch was rejected (permission denied — the owner account has no write access to the contributor's fork), so I cannot land this dedup on the PR myself.
  • Disposition: this is a minor code-cleanliness item, not a correctness blocker — the PR is mergeable as-is. The exact diff is below; the contributor can apply it before merge, or the owner can fold it in at merge time:
+/** Release an unread fetch body so its socket returns to the pool — the
+ *  #552 idle-timer contract: an abandoned body keeps its idle timer armed. */
+async function releaseUnreadBody(response: Response): Promise<void> {
+    if (!response.bodyUsed) await response.body?.cancel().catch(() => {});
+}
...
     } finally {
-        if (!result.response.bodyUsed) await result.response.body?.cancel().catch(() => {});
+        await releaseUnreadBody(result.response);
         result.stopIdleTimer();
     }

(same replacement in all 5 finally blocks)

CI: 83 check runs at 8a29d3a — 45 success / 26 skipped / 12 pending / 0 failure. master is 7 commits ahead of the PR base but touched none of this PR's files (src/web/blog-refresh.ts, tests/web-blog-refresh.test.ts, src/fetch-util.ts, src/web/blogfeed.ts) — BEHIND is not a blocker.

八问

  1. 是什么问题:blog refresh 的下载字节上限在缓冲完整响应后才生效(缺/低报 Content-Length 时超大下载跑满 EOF 才被拒),且 contents-API 回退的 base64 长度估算把 padding/换行计入文件字节、误拒合法边界文档(524287/524288 字节)。
  2. 什么原因导致的:原始下载用 arrayBuffer() 先缓冲到 EOF 再判(上限检查在缓冲之后);contents-API 用 Math.ceil(len*3/4) 估算解码长度,把 base64 的 padding 和换行也算进了文件字节数。
  3. 影响范围与严重程度:web 面板的 blog refresh 功能;特定条件下有问题(超大/边界大小的远程文档)。严重度:能用但有明显问题——上限未真正生效(内存向量)+ 边界文档被误拒;非可用性阻断、非缓存泄露,故 p2。
  4. 怎么修复的:原始下载改为流式读取(readCappedBytes,逐块对照上限、溢出即 cancel、finally releaseLock);contents-API 改为按 base64 解码后长度精确预门控(decodeCappedBase64,保留解码后守卫);5 个 fetch 函数在停止 idle watchdog 前先 cancel 未读 body(修 billion-context: 400 Bad Request when codex sends role: "developer" messages — request a request-body compat transform hook #552 泄漏)。
  5. 是否有回归问题:无。合法内容字节级不变;超大内容更早被拒(有意);边界文档 524287/524288 现在被接受(有意修复);上限值/源顺序/fail-soft 语义均未变;无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及——无缓存/重试/增量游标/去重键变更。
  7. 是否改了设计方向:不涉及——纯缺陷修复,架构/边界/数据流未转向。
  8. 是否引入了新的配置:不涉及——无新 env/config/默认值;上限值与源顺序保持不变。

判定:可合并。 修复正确、测试有效、CI 无红、无回归。唯一遗留是 5 处 cancel 行的 dedup(次要整洁项,非阻断)——我已本地验证修法(commit 79807cedbef,38/38 + typecheck 干净)但因对贡献者 fork 无写权限无法落到 PR 上,精确 diff 已给出,贡献者/owner 合并时处理即可。

Apply the review's cleanup extraction without changing the five fetch paths or the order of body cancellation and idle-watchdog shutdown. The 38 blog-refresh tests and TypeScript typecheck pass.
@Afloat16

Copy link
Copy Markdown
Contributor Author

(ai: OpenAI/GPT-6 Astra Pro)

Applied the shared releaseUnreadBody(response) helper in dcf235a. All five fetch paths still cancel an unread body before stopping the idle watchdog. The 38 blog-refresh tests and npm run typecheck pass after the extraction; the new head's full-suite verdict remains the repository CI.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Verified independently on the new head dcf235a — the ⑤b dedup finding is resolved:

  • The shared releaseUnreadBody(response) helper is correct and all five fetch paths now call it in their finally before stopIdleTimer(); no inline cancel line remains. Functionally identical to the fix I proposed (you placed it after the fetch functions and dropped the doc comment — cleaner per the "no comments unless necessary" rule).
  • npm run typecheck — clean.
  • Targeted suite tests/web-blog-refresh.test.ts — 38/38 pass on dcf235a.
  • CI on dcf235a: 43/47 checks complete (32 success, 11 skipped), 4 still in progress, 0 failures so far — the full-suite verdict lands when those finish.

The PR is now fully clean (correctness + code cleanliness). Verdict unchanged: mergeable. Nothing further needed from my side.

八问

  1. 是什么问题:上一轮 review 发现的 5 处 body-cancel 行逐字重复(⑤b 代码整洁项)。
  2. 什么原因导致的:5 个 fetch 函数各自内联了同一段 billion-context: 400 Bad Request when codex sends role: "developer" messages — request a request-body compat transform hook #552 idle-timer 清理逻辑,未抽公共。
  3. 影响范围与严重程度:仅代码整洁,无外部行为影响;非阻断。
  4. 怎么修复的:抽出 releaseUnreadBody(response) helper,5 处 finally 统一调用(行为保持)。
  5. 是否有回归问题:无——行为保持的纯重构,38/38 测试 + typecheck 在新 head 通过。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 root-cause scope/ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Blog refresh byte caps over-buffer streams and reject valid base64 documents

2 participants