Skip to content

[PR] fix: preserve caller cancellation before inline fallback - #6560

Merged
ranxianglei merged 14 commits into
ranxianglei:masterfrom
Afloat16:2026-11-13_inline-caller-cancellation
Nov 14, 2026
Merged

ranxianglei merged 14 commits into
ranxianglei:masterfrom
Afloat16:2026-11-13_inline-caller-cancellation

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #6558

Problem

makeInlineTransport distinguishes cancellation from an engine failure only by err.name === "AbortError". AbortController permits arbitrary reasons, so a cancelled generation can enter the direct upstream fallback. It can also start or dispatch work when cancellation has already happened.

Changes

Resolve the effective signal using RequestInit precedence, then call throwIfAborted before starting the engine, after startup completes and before classifying a rejected dispatch. This preserves the caller's exact reason and prevents direct retries after cancellation. Explicit signal/null overrides, transport AbortError propagation and fallback for genuine engine failure remain covered.

Validation

  • Unchanged master e1f31ad9d5b47500362699a9867819b861c7de96: 17 regression failures and 3 passing controls.
  • Fixed branch: all 20 new regression tests pass; the related pi-inline and inline-streaming suites pass, 28 tests total.
  • Coverage includes Error, TypeError, string, object and TimeoutError reasons; init/Request signal sources; cancellation before startup, during startup and during dispatch; null/replacement overrides; real engine-failure fallback.
  • Root typecheck, complete build and git diff --check pass. Hosted CI and real-model E2E remain external merge gates.

Related scope

This addresses the fallback issue identified in the review of #6504. That PR changes engine signal inheritance; #6556 changes native Request reconstruction. This change concerns the transport's cancellation/failure decision and introduces no config, kernel or persistence change.

Remote CI follow-up (2026-11-13)

CI run 43457325739 is failed, not green. I inspected every failed test job: Ubuntu Node 22/24 report the two notice-registry.test.ts assertions for the sqlite-default notice; Windows Node 22 additionally reports EBUSY while deleting index.db in persist-zstd.test.ts and persist-metadata-fold-gate.test.ts teardown. None of the changed files owns those paths. On the unchanged base e1f31ad9d5b47500362699a9867819b861c7de96, running the notice suite with the exact CI preload chain reproduces 3 passed / 2 failed (without the legacy-storage preload it is 5 passed / 0 failed).

Existing upstream PR #6561 / issue #6547 covers these same storage-baseline and Windows-handle failures. This PR keeps its focused product change separate; its full CI verdict remains failed until the upstream test fixes land and the head is rechecked. Windows reproduction here is limited to the remote job logs.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — 本 PR 修 #6558(pi-inline 传输层取消误判),与同属内联取消家族的 #6504(引擎信号继承)、#6556(native Request 重建保信号)文件不相交、机制不同,三者互补。修复本身的复现与验证已在 #6558 线程完成;本条是 PR 线程的独立审核结论。

Review (independent, head 930b66caf1, base = current master tip e1f31ad9d5, zero drift, no conflicts)

  • Diff: 2 files, +107/−0 — src/agent/pi-inline.ts +4 lines, new tests/pi-inline-cancellation.test.ts (103 lines). Clean, nothing unrelated.
  • Code: the transport resolves the effective signal once with fetch's RequestInit precedence (init.signal when defined — explicit null override included — else Request.signal), then signal?.throwIfAborted() at three decision points: before engine startup (outside the try, so a pre-startup cancel is never classified), after startup / before dispatch, and at the top of the catch before the AbortError classification. Anchoring the catch to the signal — the authoritative cancellation source — instead of pattern-matching err.name === "AbortError" is the root-cause fix: the caller's exact reason is preserved for any reason type, and a cancelled call can never reach the direct-upstream fallback or consume the one-shot degrade warning.
  • Regression audit (existing behaviors on the touched path): non-inline URL passthrough — unchanged; healthy-engine dispatch — unchanged; genuine engine failure → one-shot degrade + direct resend — preserved (pinned by the null/replacement override tests); admin-URL engine failure → throw — unchanged; transport AbortError propagation — preserved (pinned). The only behavior changes are the two intended fixes themselves (pre-aborted now rejects before the engine starts; custom-reason cancellation rejects with the exact reason, no fallback). No threshold/timing/output-format drift, no fail-fast→swallow, no config/kernel/persistence change.
  • Tests: 20 tests with teeth — 3 signal sources × 5 reason kinds (Error/TypeError/string/object/TimeoutError) assert error === reason identity AND sends === [] (no direct resend); pre-aborted asserts the engine factory is never called; mid-startup cancellation asserts dispatch never happens; the null/replacement rows are the negative direction (genuine failure still degrades — no over-suppression); the AbortError row pins the transport path. Verified in CI: 20/20 green in both shards that ran the file (ok 2100–2119, zero inline failures).

CI (verified live): 4 red shards, all attributable to non-PR causes with zero intersection with the PR diff:

Label judgment (reported, not applied — the 2026-11-12 policy forbids mutating the source PR's labels): bug / pi / scope/wiring / p2 — pi inline lane misroutes a cancelled call into the direct upstream; usable but clearly wrong under the specific condition.

八问(中文):

  1. 是什么问题:pi-inline 内联传输只认 AbortError 为取消,调用方用自定义原因(Error/TypeError/字符串/对象/TimeoutError)取消时被误判成引擎故障,在调用方已放弃请求后仍直连上游重发,并消耗一次性降级告警;已取消的请求还会照常启动引擎并 dispatch。
  2. 什么原因导致的:catch 分支用 err.name === "AbortError" 模式匹配而非以信号为取消唯一事实源;且启动前/启动后没有信号检查。
  3. 影响范围与严重程度:仅 pi 内联车道;触发条件 = 对内联模型请求的 AbortSignal 用非 AbortError 原因 abort(或启动前/启动中 abort)。严重度 p2:已取消请求被重发到上游(浪费 token、可能有副作用)+ 误触降级告警;非取消路径不受影响。
  4. 怎么修复的:传输层一次性解析"有效信号"(init.signal 优先、含 null 覆盖语义,否则 Request.signal),在启动前、启动后、catch 兜底判定前三处 throwIfAborted()——调用方取消原因原样抛出,取消请求永不进入直连兜底;真引擎故障降级与 AbortError 传播保留。
  5. 是否有回归问题:逐条审过触碰路径的既有行为——非内联直通、正常 dispatch、真故障降级(含一次性告警)、管理 URL 抛出、AbortError 传播——全部保持;唯一行为变更即本修复本身(预取消不再启动引擎;自定义取消原因不再误触发兜底),无阈值/时序/输出格式漂移,无 fail-fast 变吞错。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(单模块取消判定修正,无架构/边界/数据流转向)。
  8. 是否引入了新的配置:不涉及。

判定:代码与测试审核通过(可合)。CI 4 个红分片均为继承红/已知 flake,与本 PR diff 零交集,本 PR 自带 20 个新测试在 CI 全绿;待哨兵在 ci-fixes 收敛 visibleNotices born-red(#4599)+ 分片重跑后,由 owner 人工合并。建议 #6504/#6556 一并合入,补齐内联车道取消全链路。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI status for this PR (head 930b66caf1, run 43457325739): all 4 failing shards are inherited master-level failures, not caused by this PR's code. Verified from the live check logs this run.

Per-shard breakdown:

This PR's own 20 new tests (pi-inline-cancellation) are all green (ok 2100–2119). The PR diff (src/agent/pi-inline.ts +4 lines + new test file) has zero intersection with the failing notice/storage paths.

Why I did not push a fix to this branch: the visibleNotices hermetic fix is a master-level fix (the test lives on master, last touched by #6480). Per the CI coordination policy (owner 2026-10-24), master-level fixes MUST converge into ci-fixes (maintained by the sentinel), not be bundled into an unrelated PR branch — this PR is about pi-inline cancellation, and bundling an unrelated master-level test fix would violate one-scope + no-unrelated-changes. origin/ci-fixes is currently 0 ahead / 5 behind master — the hermetic fix has NOT converged yet, so merge origin/ci-fixes is a no-op and the branch is already based on master head e1f31ad9d5.

Blocker escalated to #4599 (CI control center): the ci-master-sentinel has been stalled since floor 1309 (spawn refused, ~8h no convergence). Multiple PRs are waiting on its ci-fixes convergence. Once the sentinel lands the visibleNotices hermetic fix in ci-fixes, this PR just needs git merge origin/ci-fixes + a re-run to go green. No change to this branch is required.


八问(中文):

  1. 是什么问题:本 PR 的 4 个 CI 失败分片全部是继承性 master 红——[PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480(pure-SQLite default)落 master 后 tests/notice-registry.test.ts 两个 visibleNotices 用例缺 storage-sqlite-default(CI preload 钉 BILI_STORAGE_UNIFIED=legacy 使 sqliteDefaultNoticeActive 门假),外加 Windows EBUSY unlink index.db 文件锁 flake 与 e2e Context Window Overflow Issue Summary: Raw Requests Sent Directly to Upstream Rejected + Local Intercept Cooldown Lockin | 上下文超窗问题汇总:请求裸冲上游被拒 + 本地拦截冷却锁死会话(统一分析入口) #4079 负载饥饿 flake。
  2. 什么原因导致的:① [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 新增/改动的存储模式测试未自行钉 BILI_STORAGE_UNIFIED,依赖生产默认值,本地过 CI 挂([PR] fix: make full-mode storage tests deterministic (#4004) #4011 教训);② Windows 文件锁(EBUSY)与负载饥饿是环境 flake,非代码缺陷。
  3. 影响范围与严重程度:仅本 PR 的 CI 检查矩阵(信号级);本 PR 自身功能(pi-inline caller cancellation)不受影响——其自带 20 个新测试 CI 全绿(ok 2100–2119),diff 与 notices/storage 零交集。
  4. 怎么修复的:本分支不动——master 级修复按 CI 协调政策必须收敛到 ci-fixes(不由本 PR 夹带,本 PR 是 pi-inline cancellation,夹带无关 master 级测试修复违反 one-scope)。已把「哨兵停滞、ci-fixes 未收敛」升级到 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599,等哨兵把 visibleNotices hermetic 修复收敛进 ci-fixes 后 merge origin/ci-fixes 重跑即绿。
  5. 是否有回归问题:不涉及(本 run 未改动任何代码)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:本 PR 代码 ready(自带测试全绿);CI 红为继承性,等 ci-fixes 收敛后重跑即绿。本分支无需任何改动。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

(ai: vllm-qwen/qwen3.8-27b)

CI verdict for this PR (head 930b66caf1, run 43457325739): all 4 failing shards are inherited master-level failures, not caused by this PR's code. Verified from the live check logs this run.

八问(中文):

  1. 是什么问题:4 个失败分片全部是继承性 master 红——[PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480(pure-SQLite default)落 master 后 tests/notice-registry.test.ts 两个 visibleNotices 用例缺 storage-sqlite-default(CI preload 钉 BILI_STORAGE_UNIFIED=legacy 使 sqliteDefaultNoticeActive 门假),外加 Windows EBUSY unlink index.db 文件锁 flake 与 e2e Context Window Overflow Issue Summary: Raw Requests Sent Directly to Upstream Rejected + Local Intercept Cooldown Lockin | 上下文超窗问题汇总:请求裸冲上游被拒 + 本地拦截冷却锁死会话(统一分析入口) #4079 负载饥饿 flake。
  2. 什么原因导致的:① [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480 存储模式测试未自行钉 BILI_STORAGE_UNIFIED,依赖生产默认值,本地过 CI 挂([PR] fix: make full-mode storage tests deterministic (#4004) #4011 教训);② Windows 文件锁(EBUSY)与负载饥饿是环境 flake,非代码缺陷。
  3. 影响范围与严重程度:仅本 PR 的 CI 检查矩阵(信号级);本 PR 功能不受影响——自带 20 个新测试 CI 全绿(ok 2100–2119),diff 与 notices/storage 零交集。
  4. 怎么修复的:本分支不动——master 级修复按 CI 协调政策必须收敛到 ci-fixes(不由本 PR 夹带,本 PR 是 pi-inline cancellation,夹带无关 master 级测试修复违反 one-scope)。已升级到 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599,等哨兵把 visibleNotices hermetic 修复收敛进 ci-fixes 后 merge origin/ci-fixes 重跑即绿。
  5. 是否有回归问题:不涉及(本 run 未改动任何代码)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:本 PR 代码 ready(自带测试全绿);CI 红为继承性,等 ci-fixes 收敛后重跑即绿。本分支无需任何改动。

The suite baseline preload (preload-storage-baseline.mjs) pins
BILI_STORAGE_UNIFIED="legacy" for the broad corpus. After ranxianglei#6480 flipped the
storage default to pure-sqlite, the notice-registry suite's visibleNotices
assertions expect the pure-sqlite default notice, which is gated on
unifiedSelectionIsSqlite (unified "sqlite" or absent) — the preload's
"legacy" made the notice silent, so the suite was born-red on master. Pin
the selection explicitly (ranxianglei#4011 lesson: a test that depends on the production
default passes locally but fails in CI).
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (ubuntu-latest, 24, 3), test (windows-latest, 22, 1), test (windows-latest, 22, 3), test (ubuntu-latest, 22, 3) — stopping for human review

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] auto-fix stalled: test (ubuntu-latest, 24, 3), test (windows-latest, 22, 1), test (windows-latest, 22, 3), test (ubuntu-latest, 22, 3) — stopping for human review

ework-agent and others added 12 commits November 14, 2026 10:20
…test

The dsh plugin lane (ci-e2e-dsh.yml) runs node --import tsx --test
WITHOUT the preload-storage-baseline.mjs preload, so BILI_STORAGE_UNIFIED
is unset and the storage mode defaults to 'full'. Under the pure-sqlite
default flip (5a80e2b, ranxianglei#6480), 'full' mode turns the JSON envelope
writer OFF — so the .content-store.json sidecar is never written, and
the test's readdirSync(namespace) hits ENOENT.

Pin BILI_STORAGE_JSON_WRITER=on in the test so the content-store
sidecar is written regardless of the storage mode. This keeps the
corrupted-store detection path exercised in both full and legacy modes.

Closes: dsh plugin lane red on master (run 43532980164, job 132255459726)
The pure-sqlite default flip (5a80e2b, ranxianglei#6480) turns the JSON envelope
writer OFF, so the .json session file this lane asserts on is never
written and 'proxy must persist the session' / 'no persisted session
carried compressed blocks' fail after the 20s wait. Pin the writer ON in
the spawned client's env (the bootstrapped proxy inherits it) so the
envelope this test was written for is still produced.

Closes: native opencode lane red on master (run 43547593803, job 132302438610)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode lane:
the JSON envelope writer is OFF, so 'proxy must persist the session
within 20000ms' fails — the on-disk .json session file is never written.
Pin the writer ON in the spawned dsh client's env (the bootstrapped
proxy inherits it).

Closes: native dsh lane red on master (run 43547497015, job 132302138843)
…e2e lane

Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode/dsh
lanes: the JSON envelope writer is OFF, so 'session persisted' fails —
the on-disk .json session file is never written. Pin the writer ON in
the spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi codex-responses WS lane red on master (run 43547626291, job 132302540960)
…lane

4th lane of the pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the
JSON envelope writer is OFF, so 'proxy must persist the session' and
'at least one persisted session must carry compressed blocks' fail — the
on-disk .json session file is never written. Pin the writer ON in the
spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi package-native lane red on ci-fixes (run 43558179054, job 132336647955)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON writer
is OFF, so the per-session CCR content-store sidecar (.content-store.json)
that 'child persists its own companion and survives a proxy restart'
asserts on is never written (0 !== 1). The earlier b1bc17b pin only
covered tests/plugin-fork.test.ts, not this file. Pin the writer ON at
module load (before any SessionStore is constructed).

Closes: dsh plugin lane red on ci-fixes (run 43558179102, job 132336519539)
…lane

Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so 'proxy must persist the session' fails — the on-disk
.json session file is never written. Pin the writer ON in the spawned
opencode client's env (the bootstrapped proxy inherits it).
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so the on-disk .json session envelope this lane asserts on
(exactly ONE persisted session under the raw key) is never written. Pin
the writer ON in the spawned dsh client's env.
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so the on-disk .json session file this lane asserts on
(state survival across the OLD→NEW update) is never written. Pin the
writer ON in the spawned proxy's env (both update phases share envBase).
@ranxianglei
ranxianglei merged commit febe718 into ranxianglei:master Nov 14, 2026
50 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit febe718f566 (Merge Afloat16/2026-11-13_inline-caller-cancellation (PR #6560)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6558 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

Review complete for PR #6560 (fix: preserve caller cancellation before inline fallback).

Verdict: APPROVE — the primary fix is correct and well-tested.

What I verified:

  • The fix correctly resolves the effective signal per RequestInit precedence (init.signal non-null > Request.signal > undefined) and adds throwIfAborted() guards before engine start, after engine start, and in the catch path (to classify caller cancellation vs. genuine engine failure).
  • All 20 tests in tests/pi-inline-cancellation.test.ts pass; the test has teeth (reverting the predicate → 1 test fails).
  • pi-inline.test.ts + inline-streaming.test.ts + pi-inline-cancellation.test.ts = 28 pass, 0 fail.
  • npx tsc --noEmit clean.
  • No kernel/ changes → no kernel-auth gate.

Non-blocking finding (latent spec deviation, not in the merged code):
The predicate init?.signal !== undefined diverges from fetch spec in exactly one case: init.signal=null + a pre-aborted Request. Per spec, only a NON-NULL init.signal overrides; null/undefined fall through to the Request's own signal. With !== undefined, signal=null masks a pre-aborted Request → guard passes → engine starts+dispatches after caller cancelled. I verified this empirically (Node fetch(req,{signal:null}) on an aborted req rejects per spec). I prepared a fix (init?.signal != null + split test) on local branch fix/pr-6560-null-signal @ 95741cf, but could not push to the contributor's fork (no write access: ranxianglei has push:False on Afloat16/billion-context). The PR was merged before this fix could be applied, so the latent deviation is still present in master. This is not triggered by any in-repo caller (no code passes init.signal=null to the inline transport), so it is latent.

PR status: Merged into master via batch PR #6677 (wave 5, merged as 3858b69, 2026-11-14). Landed via real merge commit febe718. The fix for #6558 is in master.

8问 (Chinese checklist):

  1. 是什么问题: inline transport 把调用方自定义 reason 的取消当成引擎故障,触发直连上游重试(本应直接抛错)。
  2. 什么原因导致的: makeInlineTransport 只认 err.name==='AbortError' 作为取消信号,但 AbortController 允许任意 reason(Error/TypeError/string/object/TimeoutError);且启动/分派前未查有效 signal。
  3. 影响范围与严重程度: 调用方取消后仍直连重发已放弃请求;特定条件(取消 + 自定义 reason),p2。
  4. 怎么修复的: 按 RequestInit 优先级解析有效 signal(init.signal 非 null > Request.signal > undefined),在启动前/启动后/catch 分类前三处 throwIfAborted(),保留调用方精确 reason。
  5. 是否有回归问题: 无(显式 signal/null override、transport AbortError、真引擎故障 fallback 均保持)。但发现 init.signal=null 边界(spec 偏差,latent,未触发)。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及。
  8. 是否引入了新的配置: 不涉及(仅测试 env pin BILI_STORAGE_JSON_WRITER=on)。

Conclusion: APPROVE. The primary fix is correct and the PR is now merged. The latent init.signal=null spec deviation is not triggered by any in-repo caller — dropping it per "no open ends" (not a parked debt, not a decision).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

Acknowledged — review complete, PR merged into master via batch PR #6677 (wave 5). Verdict: APPROVE. No further action needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inline transport treats custom caller cancellation as engine failure and retries direct

2 participants