Skip to content

test: issue1452 socket-hardening — replace fixed-sleep waits with deadline polling (CI flake) - #3421

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-16_flake-issue1452-stall-wait
Oct 17, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-16_flake-issue1452-stall-wait

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3420

来源: #3409 分析 PR #3409 (docs(codex) Composer-prediction eligibility gate) 时发现

What

tests/issue1452-socket-hardening.test.ts gated two assertions on async events with FIXED wall-clock sleeps, while every other wait in the same file already uses the deadline-poll idiom:

Both are converted to the file's own pattern: while (!cond && Date.now() < deadline) { await sleep(10); recheck }. The mid-stream test additionally keeps its 2s observation floor (poll up to 15s for chunk arrival, then hold until ≥2s total elapsed) so the negative assertions ("no truncation error", "no [DONE]") still cover a window far past the retired BILI_STREAM_STALL_MS=400 guard — the test's semantic claim is unchanged.

Verification

  • Targeted local run: node --import tsx --test tests/issue1452-socket-hardening.test.ts → 10/10 pass.
  • npm run typecheck clean; npm run build OK. Full-suite verdict = remote CI on this head.
  • Detection power preserved: an early-cut regression makes seen non-empty immediately (the error frame), the floor still runs to 2s, and the existing stream error / [DONE] assertions catch it exactly as before; a true hang now fails after the 15s deadline with the same message instead of at 2s.

Behavior change

None. Test-only change — no src/, config, wire, or kernel changes. No config surface touched.

Regression review

Only wait mechanics inside two tests changed; every assertion is kept verbatim. The third fixed sleep in the file (:499, 800ms) was audited and intentionally left alone — it is a semantic negative-observation window ("with the backstop knob off the socket must still be held past the moment a 300ms backstop would have fired"), where wall-clock waiting IS the assertion, not a race against an event.

…t fixed sleep)

Discovered while reviewing PR #3409: its windows-latest(22,1) shard failed
on 'retired stall guard: mid-stream silence survives any short window'
(first chunk should have flowed through) — a fixed 2s wall-clock sleep that
starves under runner load. Convert both fixed-sleep waits in this file that
gate assertions on async events to the file's own deadline-poll idiom; the
mid-stream test keeps its 2s observation floor so the negative assertions
still cover a window far past the retired 400ms guard.
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-16_flake-issue1452-stall-wait (07424ad)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4056-pr.3421.f093d4a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3421.tgz
npm install -g package

Install instructions are refreshed on each push.

…wall clock (#3435 addendum)

The deadline-poll variant still budgets the wait in Date.now() time. Client, proxy and upstream share ONE process, so a single runner freeze (observed ~42s on windows-latest in run 38930761011, issue #3435) freezes the round trip AND the deadline together: the first post-unfreeze poll iteration sees the budget expired and asserts before the poll phase can deliver the chunk. Tick counting measures process-relative progress — the same frame the retired 400ms guard lives in — so arbitrarily long freezes survive; healthy runners exit after a few ticks; a dead stream stays bounded by the tick cap plus the in-process idle-budget cut.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

来自 #3435(定时全量红灯跟踪,ce9bc90a3 上同一 flake 的第二次复现,windows-latest node 24,run 38930761011)的接力 review。已独立核对 diff 与 CI,并补推一个加固 commit 07424ad66。

结构化八问

  1. 是什么问题:tests/issue1452-socket-hardening.test.ts 两处等待(:403 首字节等待、:317 exposure 遥测行等待)用固定墙钟 sleep,Windows CI runner 负载下进程内 client→proxy→upstream 往返变慢时,墙钟窗口先耗尽、断言先触发,产生假失败(first chunk should have flowed through)。
  2. 什么原因导致的:测试的 client/proxy/upstream 同处一个进程,runner 负载或整机冻结会同时拖慢往返和 Date.now() 计时;固定墙钟预算度量的不是「进程走了多远」而是「墙上过了多久」,两者在负载下脱钩。本次实例取证:单台 windows-latest runner 出现约 42s 机器级事件循环停顿(应用内时间戳与外层时间戳偏差至 42s,整个 shard 有 10s+ 零日志窗口)。
  3. 影响范围与严重程度:仅 CI 测试确定性,无产品行为影响。严重度:定时矩阵假红灯(每次烧掉整轮矩阵时长并触发误报排查),此前两次分别伪装成潜在回归([PR] docs(codex): Codex Desktop Composer predictions — custom-provider threads are ineligible (#3408) #3409 的 win22、本次 win24)。
  4. 怎么修复的:原 PR 把两处固定 sleep 换成 deadline-poll(方向正确);我的 addendum(07424ad66)进一步把首字节等待与观察下限都改为事件循环 tick 计数(500-tick 上限等首字节、≥200-tick ≈2s 标称观察下限),因为单次 >15s 的连续冻结仍会让墙钟 deadline 在 chunk 落地前过期——微任务在每个定时器回调后、poll 阶段投递 socket 数据之前运行,解冻后的第一次 poll 迭代看到 deadline 已过期就直接断言了。tick 计量的是进程相对进度(与被退役的 400ms guard 同一时间框架):健康 runner 几个 tick 即退出;死流由 tick 上限 + BILI_UPSTREAM_TIMEOUT_MS=60000 进程内切流兜底,仍然有界失败。
  5. 是否有回归问题:纯测试文件改动,未触碰 src/;全部断言逐字保留;负断言覆盖窗口由 ≥200-tick 下限保住(≈2s 标称,与原 2s 窗口同框架);:499 那处固定 sleep 是有语义的负观察窗,有意不动(与原 PR 审计一致)。本地定向跑 10/10 通过,typecheck 干净。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及——仅测试等待机制。
  8. 是否引入了新的配置:不涉及。

CI 覆盖说明:CORE 矩阵(push/PR 触发)没有 win24 腿(只有定时全量矩阵有),所以合并后真正验证失败腿的是下一次定时全量 tick;PR head 自身 CI(含 win22×2)全绿。

判定:addendum 的 checks 转绿后即可由人工合并(合并权限在人侧)。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.498-master.076448f (rolling master tag).

npm install -g billion-context@0.0.498-master.076448f  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: fixed-sleep waits in tests/issue1452-socket-hardening.test.ts flake under CI load (first-chunk assertion false-failed on windows-latest)

1 participant