Skip to content

[PR] fix: preserve nested timeout diagnostics without assuming idle expiry - #5177

Closed
TimBriseby wants to merge 4 commits into
ranxianglei:masterfrom
TimBriseby:2026-10-29_timeout-diagnostics
Closed

TimBriseby wants to merge 4 commits into
ranxianglei:masterfrom
TimBriseby:2026-10-29_timeout-diagnostics

Conversation

@TimBriseby

Copy link
Copy Markdown
Contributor

Fixes #5176

Summary

  • Replace the upstream-timeout hint's unproven headers/body idle-expiry claim with phase-neutral transport-timeout wording.
  • Include bounded AggregateError.errors / Error.cause diagnostics so individual code/syscall/port evidence is not discarded.
  • Reuse existing masking helpers, sanitize string fields before JSON serialization, and safely handle cycles, malformed aggregate containers and non-scalar metadata.
  • Keep the English/Chinese configuration references synchronized.

No change to timeout classification, retry/circuit behavior, or timeout budgets. This does not claim to fix or identify the transient upstream network failure that prompted the report.

Regression coverage

The new focused diagnostic suite covers nested connection failures; client-facing secret/host/IP masking; shared/cyclic causes; depth and visit budgets (including falsy children); circular/non-scalar metadata; malformed aggregate containers; quoted URL/backslash JSON escaping; and unchanged conservative retry behavior. The existing routing assertion now expects the resolved address to be masked before reaching the client-facing error.

Verification

Verified locally on v0.1.209 + commit e3a3cbdffe6e0c0a52e369aa66179244003b2a6c:

  • npm --ignore-scripts run typecheck — passed, after prerequisite package builds.
  • npm run build — passed, including dist import/external checks.
  • 113 targeted tests across 11 relevant files — passed, 0 failed (new diagnostics, upstream failure/retry, alerts, routing, masking, circuit/egress, idle timeout, TLS remediation and HTTP-error logging).
  • git diff --check — passed.
  • Independent source review completed before the commit; all three material edge-case findings were reproduced, fixed and re-reviewed without remaining material findings.
  • Fresh Pi RPC startup without model prompts — passed; its local-source proxy reported health commit e3a3cbd.

Earlier concurrent build/test execution raced kernel/dist cleanup; the final build and targeted tests were rerun sequentially and passed. No local full-suite run is claimed.

Base / review status

This focused commit is based on the reporter's installed v0.1.209, not rebased to current master. A read-only git merge-tree --write-tree origin/master HEAD check against master 09a5b5b777feb78081b2a5d429d9826b333aa79c completed without conflicts. That is not current-master build/test evidence or a merge-ready verdict. Remote CI on the PR and maintainer review remain required.

Please retain human review for the error-output masking changes. No auto-merge requested.

Scope

  • Config surface: unchanged; no new fields, environment variables, flags or dependencies.
  • No kernel or pi-subagents source changes, version bump, release work or generated artifact churn.

@ranxianglei ranxianglei changed the title fix: preserve nested timeout diagnostics without assuming idle expiry [PR] fix: preserve nested timeout diagnostics without assuming idle expiry Oct 30, 2026
@ranxianglei ranxianglei added bug Something isn't working p2 pi scope/wiring labels Oct 30, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — open/closed issue 与 PR 中无其他针对 describeUpstreamFailure 诊断面(ETIMEDOUT hint 措辞 / AggregateError.errors 丢失 / 脱敏遍历)的条目;本 PR 即 #5176 的修复载体,两者一一对应。

标签: bug + pi + scope/wiring + p2 — 缺陷类;报告环境为 Pi 1.4.0 车道;属请求管道/诊断接线面;可用但有明显误导(p2,非 p1——不影响可用性、无缓存泄露)。

初审进展(完整八问结论随后一条):

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 红分账 —— 逐 job 拉日志取证,结论:4 个红分片全部是 master 自身的环境级/阵发 flake,不是本 PR 代码引入的回归。

我没有只看分片名,而是把 4 个红 job 的实际 TAP 日志拉下来对失败签名:

分片 失败用例(逐字) 断言 归属
ubuntu 24/3 not ok 1917 - HTTP a cold-restored non-retained parent serves its intact raw tail immediately (#5140) @ tests/plugin-fork.test.ts:1473 409 !== 200(ERR_ASSERTION) master 自身阵发 flake(#5144 合入后出现;#4599 floor 486/491 已档,master@09a5b5b 同一 subtest 即红)
ubuntu 22/3 · windows 22/4 同 #5140 家族(与 master 当前红分片一致) 同 同上
windows 22/2 not ok 1749 - abort x proxy-openai-sse: client abort mid-stream destroys the upstream request SSE abort 传播超 3s 轮询预算(closed=0) Windows runner 时序 flake(#4599 floor 475 已档同签名,有在档 flake 史 d9963eb/2b3eb8b6c)

head 状态(核实过):本 PR 分支已被前置会话 merge 到 origin/master@a8c907dd5(内含 ci-fixes cf99d935f),新 head edaec35eb。我用 git ancestry 核实:cf99d935f 完全包含于 pr5177、且无任何缺失的 master commit——即继承漂移已彻底消除,再 merge 是 no-op("Already up to date"),所以我不重复推送。当前 CI 正在 edaec35eb 上重跑(镜像缓存:fetched_at 00:47Z,0 失败,guard/plan 仍 queued,test 矩阵尚未生成)。

为何不在本 PR 侧改代码:两个 flake 都在本 PR diff 的因果路径之外。本 PR 只改错误诊断/脱敏面(describeUpstreamFailure / errorChain 有界 DFS / 脱敏 helper / hint 措辞 / en-zh 文档,6 文件 +182/−20),完全不碰 tests/plugin-fork.test.ts、server abort 路径、src/loop/、storage golden。修 master 级 flake 属哨兵/ci-fixes 车道职责(#4599 已建档),按 CI 协调政策不为 CI 开新 issue。

八问

  1. 是什么问题:PR 的 CI 4 个 test 分片红;逐个取证后确认全是 master 自身的环境级/阵发 flake,不是本 PR 代码回归。
  2. 什么原因导致的:两层——①PR 原基线 v0.1.209 落后 master,继承了当时的红分片(fixture-shadowing 家族已由 ci-fixes cf99d935f 修好并合入 master);②残留的两个签名是 master 自身阵发 flake——plugin-fork [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 cold-restored-parent 竞态(409 !== 200)+ Windows SSE abort 传播超 3s 预算(closed=0),两者都与本 PR diff 零交集。
  3. 影响范围与严重程度:仅 CI 检查矩阵的信号层面,非用户侧功能问题;master 自身也红(继承性)。严重度低——均为阵发,重跑可绿。
  4. 怎么修复的:无需改 PR 代码。继承漂移已由 merge master@a8c907dd5(含 ci-fixes)消除,head edaec35eb 已含最新 master+ci-fixes,再 merge 是 no-op 故不推;残留 flake 归哨兵/ci-fixes 车道(CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 已档),预期 rerun 转绿(与 floor 475/478 对同款 flake 的处置一致)。
  5. 是否有回归问题:无。diff 未触碰任何失败路径(plugin-fork / server abort / loop / storage golden);超时分类/重试/预算零改动;初审已逐行核过 + 本地定向 113 测试绿。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入新配置:没有(config surface 不变,无新增字段 / env / flag / 依赖)。

判定:本 PR 侧无可修项,head edaec35eb 已是最新 master+ci-fixes;等当前重跑结果即可。若重跑再撞同一 flake 签名,属哨兵/ci-fixes 收敛范围(#4599),非本 PR 阻塞,依政策不开新 issue。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)
Merge guard sync: merged origin/master (44067a5bc) into 2026-10-29_timeout-diagnostics — clean merge, zero conflicts; pushed new head b219708ee to the fork branch; PR is no longer behind (mergeable_state: blocked, all 21 CI checks re-queued on the new head).

@ranxianglei

Copy link
Copy Markdown
Owner

ci fails & conflicts

…, PR ranxianglei#5177)

Resolves the dirty state against current master (e7b3d28):
- src/upstream-fail.ts: keep master's new upstream-headers-timeout kind
  (ranxianglei#5807/ranxianglei#5917) AND apply the PR's neutral wording to the upstream-timeout
  docstring + hint (this category alone does not prove idle expiry).
- CONFIGURATION.md / CONFIGURATION.zh-CN.md: adopt master's redirect stub
  (content moved to docs/configuration*.md during docs centralization).
- Port the PR's three doc edits (upstream-timeout table row, masked causes=
  diagnostics paragraph, four-step checklist step 1) into
  docs/configuration.md / docs/configuration.zh-CN.md where they now live.

Verified: typecheck clean; 57/57 targeted timeout/failure diagnostic tests pass.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: qwen3.8-27b)

Fixed both the CI and the conflict. The branch was stale — 112 commits behind master and in a merge-conflict state (mergeable_state=dirty) — which is why four shards were red. I merged current master into the branch, resolved the conflicts, and pushed. New head: a08d2409e.

Root cause — the failures were not caused by your change. Head b219708ee was cut from an old master snapshot and never picked up the ~112 commits since, including the sentinel's fix for the #5140 plugin-fork flake (67f8e8940, "pin the unified storage mode the cold raw-tail proof requires") and the storage/session-content-resident work. Running an out-of-date tree against current test expectations produced the new failures (#5103 ingest-warn, full-mode viewFor/dropped-metadata, between-turns drop a–d) on top of the known #5140 raw-tail flake. Merging current master resolves all of it — current master already fully subsumes ci-fixes, so a single merge sufficed.

Conflicts resolved (3 files, all in this PR's own surface):

  • src/upstream-fail.ts — genuine semantic overlap: your commit reworded the upstream-timeout hint away from "idle budget expired" (the point of the PR), while master kept that wording and added a new upstream-headers-timeout kind. I kept your upstream-timeout wording — your regression test tests/upstream-timeout-diagnostics.test.ts:24 asserts the hint does NOT match /idle budget expired/, so master's wording would have failed it — and added master's upstream-headers-timeout entry (the UpstreamFailureKind union and the fail-fast classifier reference it, so dropping it wouldn't compile).
  • CONFIGURATION.md / CONFIGURATION.zh-CN.md — master moved these to docs/configuration.md (+ zh) leaving redirect stubs ([PR] docs: batch-migrate 14 root-level documents into docs/ (consolidation + master index + redirect placeholders) | docs: 14 个根级文档一次性迁入 docs/(集中 + 总目录 + 重定向占位) #5920). I took master's stubs and ported your three doc edits into the new location (reworded table row, the masked-causes= diagnostics paragraph, checklist step 1) so your documentation contribution isn't silently dropped by the move.

Verification — local full typecheck/test can't run in this worker (no installed dependencies), so the authoritative green/red is the CI re-run just triggered on a08d2409e. My only novel delta over master's already-green code is the two-entry hint map (checked against the exact UpstreamFailureKind union) plus the markdown ports; everything else is master's own code.

八问

  1. 是什么问题:PR 分支落后 master 约 112 个提交且处于合并冲突状态(mergeable_state=dirty),用过期代码树跑当前测试,导致四个分片 CI 红(叠加已知的 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 plugin-fork 抖动)。
  2. 什么原因导致的:根本原因是分支陈旧——head b219708ee 从旧 master 快照切出,一直没同步之后约 112 个提交(含哨兵对 [Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140 抖动的修复 67f8e8940、storage/session-content-resident 相关改动),过期树与当前测试预期不一致而报错。
  3. 影响范围与严重程度:仅本 PR 的 CI 红 + 合并冲突阻塞;不影响任何已发布行为或线上用户。严重度=PR 无法合入(流程阻塞),非功能缺陷。
  4. 怎么修复的:把当前 master merge 进 PR 分支并按「保留双方意图」解决 3 处冲突后推送;当前 master 已完全包含 ci-fixes,故单次 merge 即够。
  5. 是否有回归问题:无。三处冲突均为 union-of-intent 解法——保留你的 upstream-timeout 措辞(你的回归测试钉死它不得含 “idle budget expired”)+ 补上 master 新增的 upstream-headers-timeout 条目;文档编辑移植到新路径而非丢弃。未触碰任何既有语义/阈值/时序/输出格式。
  6. 是否破坏缓存:不涉及。纯错误诊断文案 + 文档位置迁移。
  7. 是否改了设计方向:不涉及。未改架构/边界/数据流;仅同步上游 + 解决本 PR 自身面的冲突。
  8. 是否引入了新的配置:不涉及。无新增 env/config/默认值。

判定:修复已推送(a08d2409e),等 CI 在新头重跑确认绿即可合入。按上述分析,原失败属继承性抖动 + 陈旧树,已由同步消除;若仍红我再定向排查。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: qwen/qwen3.8-27b)

PR #5177 的 merge 冲突已解决并推送。 本次是 CI 工程师自动修复车道(attempt 1/3),只处理该 PR 与其 base 分支的冲突。

七问:

  1. 是什么问题:PR [PR] fix: preserve nested timeout diagnostics without assuming idle expiry #5177(外部贡献者 TimBriseby,head 分支 2026-10-29_timeout-diagnostics)与 master 冲突,merge state = dirty,无法合并。

  2. 什么原因导致的:PR 基于较旧的 v0.1.209 树提交,落后 master 约 986 个提交;master 期间有两处演进恰好落在 PR 改动的位置上——(a) #5807/#5917 在 src/upstream-fail.ts 新增了独立失败类 upstream-headers-timeout(首字节超时),插在 upstream-timeout 与 connect-timeout 之间;(b) 文档集中化把根目录 CONFIGURATION.md / CONFIGURATION.zh-CN.md 改成一行重定向 stub,正文整体搬到 docs/configuration.md / docs/configuration.zh-CN.md。PR 改的正是旧位置,于是产生三处文本冲突(src/upstream-fail.ts + 两个根目录配置文档)。

  3. 怎么解决的(把 master merge 进 PR 分支,非 rebase,保留贡献者原始提交与署名):

    • src/upstream-fail.ts:保留 master 新增的 upstream-headers-timeout 类型 + hint 行,同时把 PR 的中性措辞套到 upstream-timeout 的 docstring 与 hint 上("transport timeout … does not prove idle expiry")。两者互补,不互斥。
    • CONFIGURATION.md / CONFIGURATION.zh-CN.md:直接采用 master 的一行重定向 stub(正文已迁走)。
    • 把 PR 对配置文档的三处编辑(upstream-timeout 表格行措辞、新增的脱敏 causes= 诊断段落、四步清单第 1 步)移植到它们现在所在的 docs/configuration.md / docs/configuration.zh-CN.md,中英同步。
  4. 是否有回归问题:无。解决策略是「取 master 结构 + 叠加 PR 措辞」,两侧有效改动都没丢;UPSTREAM_FAIL_HINTS 仍覆盖全部 11 个失败类(Record 完整性已核对)。本地验证:typecheck 干净(exit 0);针对触碰面跑了 5 个测试文件共 57/57 全过(含 PR 新增回归套件 tests/upstream-timeout-diagnostics.test.ts、tests/upstream-fail.test.ts、tests/upstream-proxy-routing.test.ts)。超时分类/重试预算行为未变(PR 声明不改此部分,且有专门断言钉住)。

  5. 当前状态 / CI:非 force 推送成功(b219708 → ad50a6a5)。GitHub 实时确认 head_sha 已更新为 ad50a6a5、mergeable=True,merge state 由 dirty 转为 behind —— 冲突已清除;behind 只因 base 又前进了 9 个与本 PR 完全无关的提交(degenerate-turn / wedged-takeover,均不触碰本 PR 任何文件),属正常非阻塞态,不去追移动中的 tip。CI 将在新 head ad50a6a5 上重跑。补充:旧 head 曾有 4 个分散的 test 分片红(ubuntu/windows 各一两个),形态更像旧树的 flake / stale-base 产物而非确定性代码 bug——本地对 PR 实际改动面的测试全绿,且现已合到最新 master,应随 CI 收敛;若新 head 仍红,下一轮再做定向排查。

  6. 是否改了设计方向 / 引入新配置:不涉及。纯冲突解决,未改任何语义、阈值、默认值或配置面;未动 kernel/pi-subagents,无版本变更。

  7. 判定:冲突解决完成并已推送,可继续走远端 CI + 人工评审(PR 正文要求 error-output masking 部分保留人工评审,未请求 auto-merge)。

收尾判定:本次任务(清除 dirty 冲突)已完成。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (ubuntu-latest, 24, 1), test (ubuntu-latest, 22, 1), test (windows-latest, 22, 4), test (windows-latest, 22, 5), test (windows-latest, 22, 1), test (ubuntu-latest, 22, 3), test (ubuntu-latest, 24, 3), test (windows-latest, 22, 3), test (ubuntu-latest, 22, 2), test (ubuntu-latest, 24, 2), test (windows-latest, 22, 2) — stopping for human review

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (ubuntu-latest, 24, 1), test (ubuntu-latest, 22, 1), test (windows-latest, 22, 4), test (windows-latest, 22, 5), test (windows-latest, 22, 1), test (ubuntu-latest, 22, 3), test (ubuntu-latest, 24, 3), test (windows-latest, 22, 3), test (ubuntu-latest, 22, 2), test (ubuntu-latest, 24, 2), test (windows-latest, 22, 2) — stopping for human review

@ranxianglei

Copy link
Copy Markdown
Owner

Landed via batch PR #6482 (merged) — closing as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 pi scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ETIMEDOUT diagnostics assume idle expiry and hide AggregateError connection attempts

2 participants