Skip to content

Commit be37baf

Browse files
committed
fix(orch): mark paused-at-gate sessions as awaiting_input (#42)
The v2.0.0-rc3 finalizer-asymmetry fix (PR #41) correctly handled graph-completed-without-pause: every non-streaming run now calls _finalize_session_status_async() to flip the row to a terminal status. But the *paused* branch was left as a status-write no-op: when the graph paused at a HITL gate, the session row stayed at its pre-pause status ('new' / 'in_progress') instead of transitioning to 'awaiting_input'. UIs that filter by status='awaiting_input' (the approvals queue, the sessions rail Active group, /sessions in-flight listing) missed paused sessions entirely. This was the sibling case of CRITICAL #1 and was filed as #42 after the rc3 ship. The fix adds a new lock-guarded helper that mirrors _finalize_session_status_async on the paused side: Orchestrator._mark_session_paused_async(session_id) -> str | None It loads the row, captures the from-status, flips status to 'awaiting_input', saves, and emits both status_changed (per-session event log) and session.status_changed (cross-session SSE) events via the existing _emit_status_changed_event helper. It is a no-op when the row is already at 'awaiting_input', already at a terminal status (guard against a late paused-write unwinding a finalize that landed in between), or the row is missing. Four call sites updated to use it: the three start_session + stream_session + retry_session paths in orchestrator.py, and the OrchestratorService._run() background task in service.py. Regression coverage in tests/test_finalizer_paths.py: - _PausedAtGateGraph fake graph that returns from ainvoke and reports a non-empty `next` tuple to aget_state (mirrors langgraph 1.x's interrupt-boundary state). - test_orchestrator_start_session_marks_paused_run_awaiting_input: direct (non-streaming) path produces status='awaiting_input' on the row AND emits status_changed + session.status_changed events. - test_service_background_start_session_marks_paused_run_awaiting_input: same guarantee through the OrchestratorService._run() task. - test_mark_session_paused_is_no_op_when_already_awaiting_input: no spurious event emission. - test_mark_session_paused_is_no_op_on_terminal_status: guard against a late paused-write unwinding a completed terminal flip. Two pre-existing shim tests updated to stub the new method: - tests/test_retry_session_locked_post_policy.py _StubOrch - tests/test_triggers/test_orchestrator_trigger_kwarg.py (orchestrator built via __new__, bypasses __init__) Verified: - ASR_WEB_DIST=/tmp/empty uv run pytest --no-cov -> 1349 passed, 8 skipped - uv run ruff check src/ tests/ -> clean - uv run pyright src/runtime -> clean - cd web && npm run typecheck/lint/test:unit/build/check:size -> green - dist/* regenerated (HARD-08)
1 parent 8be7ea2 commit be37baf

8 files changed

Lines changed: 411 additions & 12 deletions

File tree

dist/app.py

Lines changed: 62 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6458,7 +6458,11 @@ async def _run() -> None:
64586458
),
64596459
config=orch._thread_config(session_id),
64606460
)
6461-
if not await orch._is_graph_paused(session_id):
6461+
if await orch._is_graph_paused(session_id):
6462+
# Issue #42: mark the row awaiting_input so UIs
6463+
# filtering by that status see the paused session.
6464+
await orch._mark_session_paused_async(session_id)
6465+
else:
64626466
await orch._finalize_session_status_async(session_id)
64636467
except asyncio.CancelledError:
64646468
raise
@@ -14882,6 +14886,54 @@ async def _finalize_session_status_async(
1488214886
async with self._locks.acquire(session_id):
1488314887
return self._finalize_session_status(session_id)
1488414888

14889+
async def _mark_session_paused_async(
14890+
self, session_id: str,
14891+
) -> str | None:
14892+
"""Lock-guarded write that flips a paused session to
14893+
``"awaiting_input"`` and emits the matching ``status_changed``
14894+
events.
14895+
14896+
Companion to :meth:`_finalize_session_status_async`. The
14897+
finalizer handles graph-completed runs; this handles
14898+
graph-paused runs (HITL gate). Without it, sessions that
14899+
pause at a gate keep their old status (typically ``"new"`` /
14900+
``"in_progress"``) and UIs that filter by ``"awaiting_input"``
14901+
(the approvals queue, the rail's Active group) miss them — see
14902+
issue #42.
14903+
14904+
No-op (returns ``None``) when the session is already at
14905+
``"awaiting_input"``, already in a terminal status (a late
14906+
pause-write must not unwind a finalize that landed in
14907+
between), or the row is missing. Otherwise transitions the
14908+
row to ``"awaiting_input"`` and emits both per-session
14909+
``status_changed`` and cross-session
14910+
``session.status_changed`` events via
14911+
:func:`_emit_status_changed_event`.
14912+
"""
14913+
async with self._locks.acquire(session_id):
14914+
try:
14915+
inc = self.store.load(session_id)
14916+
except (FileNotFoundError, ValueError, KeyError, LookupError):
14917+
return None
14918+
if inc.status == "awaiting_input":
14919+
return None
14920+
statuses = getattr(getattr(self, "cfg", None), "orchestrator", None)
14921+
if statuses is not None:
14922+
current_def = statuses.statuses.get(inc.status)
14923+
if current_def is not None and current_def.terminal:
14924+
return None
14925+
from_status = inc.status
14926+
inc.status = "awaiting_input"
14927+
self.store.save(inc)
14928+
_emit_status_changed_event(
14929+
orch=self,
14930+
inc=inc,
14931+
from_status=from_status,
14932+
to_status="awaiting_input",
14933+
cause="gate_paused",
14934+
)
14935+
return "awaiting_input"
14936+
1488514937
async def _is_graph_paused(self, session_id: str) -> bool:
1488614938
"""Return True iff the compiled graph has a pending step waiting
1488714939
to resume (i.e. it's paused at an ``interrupt()`` boundary).
@@ -15082,7 +15134,9 @@ async def start_session(self, *, query: str,
1508215134
last_agent=None, error=None),
1508315135
config=self._thread_config(inc.id),
1508415136
)
15085-
if not await self._is_graph_paused(inc.id):
15137+
if await self._is_graph_paused(inc.id):
15138+
await self._mark_session_paused_async(inc.id)
15139+
else:
1508615140
await self._finalize_session_status_async(inc.id)
1508715141
return inc.id
1508815142

@@ -15140,7 +15194,10 @@ async def stream_session(self, *, query: str, environment: str,
1514015194
# session is waiting for operator approval, not done. Stamping
1514115195
# default_terminal_status here would orphan the pending_approval
1514215196
# ToolCall row written by the gateway just before the pause.
15197+
# Issue #42: still flip the session row to awaiting_input so
15198+
# the approvals queue + sessions rail Active group pick it up.
1514315199
if await self._is_graph_paused(inc.id):
15200+
await self._mark_session_paused_async(inc.id)
1514415201
yield {"event": "session_paused", "incident_id": inc.id,
1514515202
"ts": _event_ts()}
1514615203
else:
@@ -15422,8 +15479,10 @@ async def _retry_session_locked(self, session_id: str) -> AsyncIterator[dict]:
1542215479
yield self._to_ui_event(ev, session_id)
1542315480
# See ``stream_session`` for why pause-detection guards the
1542415481
# finalize call: a HITL pause must not be coerced into a
15425-
# terminal status.
15482+
# terminal status. Issue #42: still flip to awaiting_input so
15483+
# UIs filtering by that status see the retried session.
1542615484
if await self._is_graph_paused(session_id):
15485+
await self._mark_session_paused_async(session_id)
1542715486
yield {"event": "session_paused", "incident_id": session_id,
1542815487
"ts": _event_ts()}
1542915488
else:

dist/apps/code-review.py

Lines changed: 62 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6511,7 +6511,11 @@ async def _run() -> None:
65116511
),
65126512
config=orch._thread_config(session_id),
65136513
)
6514-
if not await orch._is_graph_paused(session_id):
6514+
if await orch._is_graph_paused(session_id):
6515+
# Issue #42: mark the row awaiting_input so UIs
6516+
# filtering by that status see the paused session.
6517+
await orch._mark_session_paused_async(session_id)
6518+
else:
65156519
await orch._finalize_session_status_async(session_id)
65166520
except asyncio.CancelledError:
65176521
raise
@@ -14935,6 +14939,54 @@ async def _finalize_session_status_async(
1493514939
async with self._locks.acquire(session_id):
1493614940
return self._finalize_session_status(session_id)
1493714941

14942+
async def _mark_session_paused_async(
14943+
self, session_id: str,
14944+
) -> str | None:
14945+
"""Lock-guarded write that flips a paused session to
14946+
``"awaiting_input"`` and emits the matching ``status_changed``
14947+
events.
14948+
14949+
Companion to :meth:`_finalize_session_status_async`. The
14950+
finalizer handles graph-completed runs; this handles
14951+
graph-paused runs (HITL gate). Without it, sessions that
14952+
pause at a gate keep their old status (typically ``"new"`` /
14953+
``"in_progress"``) and UIs that filter by ``"awaiting_input"``
14954+
(the approvals queue, the rail's Active group) miss them — see
14955+
issue #42.
14956+
14957+
No-op (returns ``None``) when the session is already at
14958+
``"awaiting_input"``, already in a terminal status (a late
14959+
pause-write must not unwind a finalize that landed in
14960+
between), or the row is missing. Otherwise transitions the
14961+
row to ``"awaiting_input"`` and emits both per-session
14962+
``status_changed`` and cross-session
14963+
``session.status_changed`` events via
14964+
:func:`_emit_status_changed_event`.
14965+
"""
14966+
async with self._locks.acquire(session_id):
14967+
try:
14968+
inc = self.store.load(session_id)
14969+
except (FileNotFoundError, ValueError, KeyError, LookupError):
14970+
return None
14971+
if inc.status == "awaiting_input":
14972+
return None
14973+
statuses = getattr(getattr(self, "cfg", None), "orchestrator", None)
14974+
if statuses is not None:
14975+
current_def = statuses.statuses.get(inc.status)
14976+
if current_def is not None and current_def.terminal:
14977+
return None
14978+
from_status = inc.status
14979+
inc.status = "awaiting_input"
14980+
self.store.save(inc)
14981+
_emit_status_changed_event(
14982+
orch=self,
14983+
inc=inc,
14984+
from_status=from_status,
14985+
to_status="awaiting_input",
14986+
cause="gate_paused",
14987+
)
14988+
return "awaiting_input"
14989+
1493814990
async def _is_graph_paused(self, session_id: str) -> bool:
1493914991
"""Return True iff the compiled graph has a pending step waiting
1494014992
to resume (i.e. it's paused at an ``interrupt()`` boundary).
@@ -15135,7 +15187,9 @@ async def start_session(self, *, query: str,
1513515187
last_agent=None, error=None),
1513615188
config=self._thread_config(inc.id),
1513715189
)
15138-
if not await self._is_graph_paused(inc.id):
15190+
if await self._is_graph_paused(inc.id):
15191+
await self._mark_session_paused_async(inc.id)
15192+
else:
1513915193
await self._finalize_session_status_async(inc.id)
1514015194
return inc.id
1514115195

@@ -15193,7 +15247,10 @@ async def stream_session(self, *, query: str, environment: str,
1519315247
# session is waiting for operator approval, not done. Stamping
1519415248
# default_terminal_status here would orphan the pending_approval
1519515249
# ToolCall row written by the gateway just before the pause.
15250+
# Issue #42: still flip the session row to awaiting_input so
15251+
# the approvals queue + sessions rail Active group pick it up.
1519615252
if await self._is_graph_paused(inc.id):
15253+
await self._mark_session_paused_async(inc.id)
1519715254
yield {"event": "session_paused", "incident_id": inc.id,
1519815255
"ts": _event_ts()}
1519915256
else:
@@ -15475,8 +15532,10 @@ async def _retry_session_locked(self, session_id: str) -> AsyncIterator[dict]:
1547515532
yield self._to_ui_event(ev, session_id)
1547615533
# See ``stream_session`` for why pause-detection guards the
1547715534
# finalize call: a HITL pause must not be coerced into a
15478-
# terminal status.
15535+
# terminal status. Issue #42: still flip to awaiting_input so
15536+
# UIs filtering by that status see the retried session.
1547915537
if await self._is_graph_paused(session_id):
15538+
await self._mark_session_paused_async(session_id)
1548015539
yield {"event": "session_paused", "incident_id": session_id,
1548115540
"ts": _event_ts()}
1548215541
else:

dist/apps/incident-management.py

Lines changed: 62 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6523,7 +6523,11 @@ async def _run() -> None:
65236523
),
65246524
config=orch._thread_config(session_id),
65256525
)
6526-
if not await orch._is_graph_paused(session_id):
6526+
if await orch._is_graph_paused(session_id):
6527+
# Issue #42: mark the row awaiting_input so UIs
6528+
# filtering by that status see the paused session.
6529+
await orch._mark_session_paused_async(session_id)
6530+
else:
65276531
await orch._finalize_session_status_async(session_id)
65286532
except asyncio.CancelledError:
65296533
raise
@@ -14947,6 +14951,54 @@ async def _finalize_session_status_async(
1494714951
async with self._locks.acquire(session_id):
1494814952
return self._finalize_session_status(session_id)
1494914953

14954+
async def _mark_session_paused_async(
14955+
self, session_id: str,
14956+
) -> str | None:
14957+
"""Lock-guarded write that flips a paused session to
14958+
``"awaiting_input"`` and emits the matching ``status_changed``
14959+
events.
14960+
14961+
Companion to :meth:`_finalize_session_status_async`. The
14962+
finalizer handles graph-completed runs; this handles
14963+
graph-paused runs (HITL gate). Without it, sessions that
14964+
pause at a gate keep their old status (typically ``"new"`` /
14965+
``"in_progress"``) and UIs that filter by ``"awaiting_input"``
14966+
(the approvals queue, the rail's Active group) miss them — see
14967+
issue #42.
14968+
14969+
No-op (returns ``None``) when the session is already at
14970+
``"awaiting_input"``, already in a terminal status (a late
14971+
pause-write must not unwind a finalize that landed in
14972+
between), or the row is missing. Otherwise transitions the
14973+
row to ``"awaiting_input"`` and emits both per-session
14974+
``status_changed`` and cross-session
14975+
``session.status_changed`` events via
14976+
:func:`_emit_status_changed_event`.
14977+
"""
14978+
async with self._locks.acquire(session_id):
14979+
try:
14980+
inc = self.store.load(session_id)
14981+
except (FileNotFoundError, ValueError, KeyError, LookupError):
14982+
return None
14983+
if inc.status == "awaiting_input":
14984+
return None
14985+
statuses = getattr(getattr(self, "cfg", None), "orchestrator", None)
14986+
if statuses is not None:
14987+
current_def = statuses.statuses.get(inc.status)
14988+
if current_def is not None and current_def.terminal:
14989+
return None
14990+
from_status = inc.status
14991+
inc.status = "awaiting_input"
14992+
self.store.save(inc)
14993+
_emit_status_changed_event(
14994+
orch=self,
14995+
inc=inc,
14996+
from_status=from_status,
14997+
to_status="awaiting_input",
14998+
cause="gate_paused",
14999+
)
15000+
return "awaiting_input"
15001+
1495015002
async def _is_graph_paused(self, session_id: str) -> bool:
1495115003
"""Return True iff the compiled graph has a pending step waiting
1495215004
to resume (i.e. it's paused at an ``interrupt()`` boundary).
@@ -15147,7 +15199,9 @@ async def start_session(self, *, query: str,
1514715199
last_agent=None, error=None),
1514815200
config=self._thread_config(inc.id),
1514915201
)
15150-
if not await self._is_graph_paused(inc.id):
15202+
if await self._is_graph_paused(inc.id):
15203+
await self._mark_session_paused_async(inc.id)
15204+
else:
1515115205
await self._finalize_session_status_async(inc.id)
1515215206
return inc.id
1515315207

@@ -15205,7 +15259,10 @@ async def stream_session(self, *, query: str, environment: str,
1520515259
# session is waiting for operator approval, not done. Stamping
1520615260
# default_terminal_status here would orphan the pending_approval
1520715261
# ToolCall row written by the gateway just before the pause.
15262+
# Issue #42: still flip the session row to awaiting_input so
15263+
# the approvals queue + sessions rail Active group pick it up.
1520815264
if await self._is_graph_paused(inc.id):
15265+
await self._mark_session_paused_async(inc.id)
1520915266
yield {"event": "session_paused", "incident_id": inc.id,
1521015267
"ts": _event_ts()}
1521115268
else:
@@ -15487,8 +15544,10 @@ async def _retry_session_locked(self, session_id: str) -> AsyncIterator[dict]:
1548715544
yield self._to_ui_event(ev, session_id)
1548815545
# See ``stream_session`` for why pause-detection guards the
1548915546
# finalize call: a HITL pause must not be coerced into a
15490-
# terminal status.
15547+
# terminal status. Issue #42: still flip to awaiting_input so
15548+
# UIs filtering by that status see the retried session.
1549115549
if await self._is_graph_paused(session_id):
15550+
await self._mark_session_paused_async(session_id)
1549215551
yield {"event": "session_paused", "incident_id": session_id,
1549315552
"ts": _event_ts()}
1549415553
else:

src/runtime/orchestrator.py

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1091,6 +1091,54 @@ async def _finalize_session_status_async(
10911091
async with self._locks.acquire(session_id):
10921092
return self._finalize_session_status(session_id)
10931093

1094+
async def _mark_session_paused_async(
1095+
self, session_id: str,
1096+
) -> str | None:
1097+
"""Lock-guarded write that flips a paused session to
1098+
``"awaiting_input"`` and emits the matching ``status_changed``
1099+
events.
1100+
1101+
Companion to :meth:`_finalize_session_status_async`. The
1102+
finalizer handles graph-completed runs; this handles
1103+
graph-paused runs (HITL gate). Without it, sessions that
1104+
pause at a gate keep their old status (typically ``"new"`` /
1105+
``"in_progress"``) and UIs that filter by ``"awaiting_input"``
1106+
(the approvals queue, the rail's Active group) miss them — see
1107+
issue #42.
1108+
1109+
No-op (returns ``None``) when the session is already at
1110+
``"awaiting_input"``, already in a terminal status (a late
1111+
pause-write must not unwind a finalize that landed in
1112+
between), or the row is missing. Otherwise transitions the
1113+
row to ``"awaiting_input"`` and emits both per-session
1114+
``status_changed`` and cross-session
1115+
``session.status_changed`` events via
1116+
:func:`_emit_status_changed_event`.
1117+
"""
1118+
async with self._locks.acquire(session_id):
1119+
try:
1120+
inc = self.store.load(session_id)
1121+
except (FileNotFoundError, ValueError, KeyError, LookupError):
1122+
return None
1123+
if inc.status == "awaiting_input":
1124+
return None
1125+
statuses = getattr(getattr(self, "cfg", None), "orchestrator", None)
1126+
if statuses is not None:
1127+
current_def = statuses.statuses.get(inc.status)
1128+
if current_def is not None and current_def.terminal:
1129+
return None
1130+
from_status = inc.status
1131+
inc.status = "awaiting_input"
1132+
self.store.save(inc)
1133+
_emit_status_changed_event(
1134+
orch=self,
1135+
inc=inc,
1136+
from_status=from_status,
1137+
to_status="awaiting_input",
1138+
cause="gate_paused",
1139+
)
1140+
return "awaiting_input"
1141+
10941142
async def _is_graph_paused(self, session_id: str) -> bool:
10951143
"""Return True iff the compiled graph has a pending step waiting
10961144
to resume (i.e. it's paused at an ``interrupt()`` boundary).
@@ -1291,7 +1339,9 @@ async def start_session(self, *, query: str,
12911339
last_agent=None, error=None),
12921340
config=self._thread_config(inc.id),
12931341
)
1294-
if not await self._is_graph_paused(inc.id):
1342+
if await self._is_graph_paused(inc.id):
1343+
await self._mark_session_paused_async(inc.id)
1344+
else:
12951345
await self._finalize_session_status_async(inc.id)
12961346
return inc.id
12971347

@@ -1349,7 +1399,10 @@ async def stream_session(self, *, query: str, environment: str,
13491399
# session is waiting for operator approval, not done. Stamping
13501400
# default_terminal_status here would orphan the pending_approval
13511401
# ToolCall row written by the gateway just before the pause.
1402+
# Issue #42: still flip the session row to awaiting_input so
1403+
# the approvals queue + sessions rail Active group pick it up.
13521404
if await self._is_graph_paused(inc.id):
1405+
await self._mark_session_paused_async(inc.id)
13531406
yield {"event": "session_paused", "incident_id": inc.id,
13541407
"ts": _event_ts()}
13551408
else:
@@ -1631,8 +1684,10 @@ async def _retry_session_locked(self, session_id: str) -> AsyncIterator[dict]:
16311684
yield self._to_ui_event(ev, session_id)
16321685
# See ``stream_session`` for why pause-detection guards the
16331686
# finalize call: a HITL pause must not be coerced into a
1634-
# terminal status.
1687+
# terminal status. Issue #42: still flip to awaiting_input so
1688+
# UIs filtering by that status see the retried session.
16351689
if await self._is_graph_paused(session_id):
1690+
await self._mark_session_paused_async(session_id)
16361691
yield {"event": "session_paused", "incident_id": session_id,
16371692
"ts": _event_ts()}
16381693
else:

0 commit comments

Comments
 (0)