fix(gateway): close runner stream on turn exit and tolerate cross-Context scope teardown - #1570
Open
xiaohuzai wants to merge 2 commits into
Open
fix(gateway): close runner stream on turn exit and tolerate cross-Context scope teardown#1570xiaohuzai wants to merge 2 commits into
xiaohuzai wants to merge 2 commits into
Conversation
added 2 commits
September 5, 2026 15:55
…text scope teardown Every chat.abort left the turn's run generator to asyncio's async-generator finalizer, which runs aclose() in a fresh Context where the run generator's scope stack cannot reset its ContextVar tokens: each abort logged "Task exception was never retrieved" with a nested ValueError chain, and the subscriber-visible turn-terminal event could be lost (TokenRhythm#1569). - run_direct_turn closes the composed runner stream in its finally so the scope stack unwinds in the Context that entered it - the five turn scopes (process ownership, sandbox policy, Git run mode, runtime pack, managed toolchain state) reset their tokens through a helper that tolerates cross-Context teardown as a backstop Verified on a live loopback gateway: 5 aborts -> 5 orphan crashes on main, 0 crashes after this change, terminal events delivered on every abort.
CI on TokenRhythm#1570 failed on the repo-wide ruff check (import order in the new finalizer regression test) and would have failed mypy: wrap_stream returns AsyncIterator, whose protocol has no aclose — use the same getattr idiom stream_wrappers.py already uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Fix the deterministic
chat.abortcleanup crash (#1569): every abort left theturn's
TurnRunner.runasync generator to asyncio's async-generator finalizer,which runs
aclose()in a fresh asyncio Context. The run generator iswrapped in a five-scope
withstack (managed toolchain state, runtime packstate, sandbox policy, Git run mode, task process scope), and each scope's
finallyresets aContextVartoken issued in the creating Context —ContextVar.reset()raisesValueErrorcross-Context, so every abort loggedTask exception was never retrievedwith a 6-segment nested ValueError chain(immediately after
task_runtime.cancellation_settled reason='user_abort').Depending on timing the exception chain could also swallow the
subscriber-visible turn-terminal event, leaving third-party WebSocket clients
unable to distinguish "cancelled" from "still running" without a timeout.
Two complementary changes:
run_direct_turnnow closes thecomposed runner stream in its
finally. Unwinding happens in the Contextthat entered the scope stack (the consumer, or the heartbeat driver the
consumer settles), so the generator never reaches the GC finalizer on the
normal paths.
their tokens through a shared
reset_contextvar_tokenhelper that swallowsexactly the cross-Context
ValueError, so a generator finalized by the GChook can no longer crash regardless of who closes it. The issuing Context's
value is untouched by a foreign reset (documented in the helper and locked
in by a test).
Scope boundary:
run_direct_turn+ the five scope context managers + a sharedhelper in
asyncio_utils.Non-goals: no changes to
chat.abortRPC semantics, reply payloads, eventnames, or the wrapper composition order in
wrap_stream.Branch
Base branch: main
Target exception: N/A
Issue
Linked issue: Fixes #1569
Release Note
Release note: Aborting a turn no longer crashes the gateway's cleanup path
(
Task exception was never retrieved/ValueError: ... was created in a different Contexton everychat.abort), and the turn-terminal event is nowreliably delivered to subscribed clients.
Tests
Ruff: all changed files pass
ruff check.Pytest (offline, deterministic, credential-free):
tests/test_asyncio_utils.py— 4 passedtests/test_gateway/test_direct_turn_runtime.py— 3 passed (incl. new abortregression test)
tests/test_engine/test_turn_scope_stack_finalizer.py— 1 passed (new: realfive-scope stack + GC-finalizer teardown; fails on unpatched scopes with
the exact production
ValueErrorsignature)tests/test_process_tree.py,tests/test_git_runtime.py,tests/test_skills/test_toolchain_state_scope.py,tests/test_skills/test_skill_exec_non_blocking.py,tests/test_skills/test_toolchain_runtime_integration.py,tests/test_runtime_packs/test_runtime_pack_manager.py,tests/test_sandbox/test_inprocess_managed_network.py,tests/test_gateway/test_p1a_exact_abort_contract.py— 227 passed, 7 skippedBuild: N/A
Regression tests: added
Notes: additionally verified end-to-end on a live loopback gateway (source
checkout, streaming provider): 5 sequential aborts → 5 orphan-task crashes on
unpatched
main@94ac35eb9; 8 sequential aborts → 0 crashes with thischange, with
task.cancelleddelivered to the subscriber on every abort.Maintainer Live Check
Maintainer live check: no
Surface: gateway — the unit tests reproduce the crash mechanism offline; a
live check would be
chat.senda long streaming answer +chat.abortfromany WS client and observe the gateway log.
Safety
No secrets, local-only artifacts, private prompts/transcripts, channel
identifiers, AI session artifacts, or
tests/_private/contents are included.Third-Party Origin
Third-party origin: none
Documentation Changes