Skip to content

fix(gateway): close runner stream on turn exit and tolerate cross-Context scope teardown - #1570

Open
xiaohuzai wants to merge 2 commits into
TokenRhythm:mainfrom
xiaohuzai:fix/abort-contextvar-cleanup
Open

fix(gateway): close runner stream on turn exit and tolerate cross-Context scope teardown#1570
xiaohuzai wants to merge 2 commits into
TokenRhythm:mainfrom
xiaohuzai:fix/abort-contextvar-cleanup

Conversation

@xiaohuzai

Copy link
Copy Markdown
Contributor

Scope

Fix the deterministic chat.abort cleanup crash (#1569): every abort left the
turn's TurnRunner.run async generator to asyncio's async-generator finalizer,
which runs aclose() in a fresh asyncio Context. The run generator is
wrapped in a five-scope with stack (managed toolchain state, runtime pack
state, sandbox policy, Git run mode, task process scope), and each scope's
finally resets a ContextVar token issued in the creating Context —
ContextVar.reset() raises ValueError cross-Context, so every abort logged
Task exception was never retrieved with a 6-segment nested ValueError chain
(immediately after task_runtime.cancellation_settled reason='user_abort').
Depending on timing the exception chain could also swallow the
subscriber-visible turn-terminal event, leaving third-party WebSocket clients
unable to distinguish "cancelled" from "still running" without a timeout.

Two complementary changes:

  1. Deterministic close (root cause)run_direct_turn now closes the
    composed runner stream in its finally. Unwinding happens in the Context
    that entered the scope stack (the consumer, or the heartbeat driver the
    consumer settles), so the generator never reaches the GC finalizer on the
    normal paths.
  2. Cross-Context-tolerant teardown (backstop) — the five turn scopes reset
    their tokens through a shared reset_contextvar_token helper that swallows
    exactly the cross-Context ValueError, so a generator finalized by the GC
    hook can no longer crash regardless of who closes it. The issuing Context's
    value is untouched by a foreign reset (documented in the helper and locked
    in by a test).

Scope boundary: run_direct_turn + the five scope context managers + a shared
helper in asyncio_utils.

Non-goals: no changes to chat.abort RPC semantics, reply payloads, event
names, or the wrapper composition order in wrap_stream.

Branch

Base branch: main

Target exception: N/A

Issue

Linked issue: Fixes #1569

Release Note

Release note: Aborting a turn no longer crashes the gateway's cleanup path
(Task exception was never retrieved / ValueError: ... was created in a different Context on every chat.abort), and the turn-terminal event is now
reliably delivered to subscribed clients.

Tests

Ruff: all changed files pass ruff check.

Pytest (offline, deterministic, credential-free):

  • tests/test_asyncio_utils.py — 4 passed
  • tests/test_gateway/test_direct_turn_runtime.py — 3 passed (incl. new abort
    regression test)
  • tests/test_engine/test_turn_scope_stack_finalizer.py — 1 passed (new: real
    five-scope stack + GC-finalizer teardown; fails on unpatched scopes with
    the exact production ValueError signature
    )
  • touched scope suites: tests/test_process_tree.py,
    tests/test_git_runtime.py, tests/test_skills/test_toolchain_state_scope.py,
    tests/test_skills/test_skill_exec_non_blocking.py,
    tests/test_skills/test_toolchain_runtime_integration.py,
    tests/test_runtime_packs/test_runtime_pack_manager.py,
    tests/test_sandbox/test_inprocess_managed_network.py,
    tests/test_gateway/test_p1a_exact_abort_contract.py — 227 passed, 7 skipped

Build: N/A

Regression tests: added

Notes: additionally verified end-to-end on a live loopback gateway (source
checkout, streaming provider): 5 sequential aborts → 5 orphan-task crashes on
unpatched main @ 94ac35eb9; 8 sequential aborts → 0 crashes with this
change, with task.cancelled delivered to the subscriber on every abort.

Maintainer Live Check

Maintainer live check: no

Surface: gateway — the unit tests reproduce the crash mechanism offline; a
live check would be chat.send a long streaming answer + chat.abort from
any WS client and observe the gateway log.

Safety

No secrets, local-only artifacts, private prompts/transcripts, channel
identifiers, AI session artifacts, or tests/_private/ contents are included.

Third-Party Origin

Third-party origin: none

Documentation Changes

  • Links point to existing repository files or stable external pages.
  • Code fences and Markdown tables render correctly on GitHub.
  • Examples avoid real secrets, local private paths, and private transcripts.

Hermes Agent added 2 commits September 5, 2026 15:55
…text scope teardown

Every chat.abort left the turn's run generator to asyncio's async-generator
finalizer, which runs aclose() in a fresh Context where the run generator's
scope stack cannot reset its ContextVar tokens: each abort logged
"Task exception was never retrieved" with a nested ValueError chain, and
the subscriber-visible turn-terminal event could be lost (TokenRhythm#1569).

- run_direct_turn closes the composed runner stream in its finally so the
  scope stack unwinds in the Context that entered it
- the five turn scopes (process ownership, sandbox policy, Git run mode,
  runtime pack, managed toolchain state) reset their tokens through a
  helper that tolerates cross-Context teardown as a backstop

Verified on a live loopback gateway: 5 aborts -> 5 orphan crashes on main,
0 crashes after this change, terminal events delivered on every abort.
CI on TokenRhythm#1570 failed on the repo-wide ruff check (import order in the new
finalizer regression test) and would have failed mypy: wrap_stream returns
AsyncIterator, whose protocol has no aclose — use the same getattr idiom
stream_wrappers.py already uses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: chat.abort leaves an orphan async-generator finalizer task crashing on cross-Context ContextVar resets (can drop turn-terminal events)

1 participant