Skip to content

fix(runner): keep stderr chatter from suppressing idle watchdogs - #108

Open
Haloukaidi wants to merge 1 commit into
lbx154:devfrom
Haloukaidi:fix/stderr-idle-watchdog
Open

Haloukaidi wants to merge 1 commit into
lbx154:devfrom
Haloukaidi:fix/stderr-idle-watchdog

Conversation

@Haloukaidi

Copy link
Copy Markdown

Continuous stderr output currently resets the activity clock and keeps the queue nonempty, so idle watchdog checks can be postponed indefinitely even when stdout has stopped progressing.

Keep the existing stdout activity clock independent of diagnostic stderr, and run the same idle checks while processing stderr as well as while the queue is empty. Diagnostic capture and callbacks remain available; normal stdout still resets escalation. The default warning-only policy is unchanged.

Validation on Windows / Python 3.13:

  • 45 passed, 4 platform-specific skips across the stream callback, idle watchdog, and execution-host failure modules.
  • Five real-pipe regression cases cover stderr chatter, an explicit hard deadline, inactivity callbacks, stdout resetting warning escalation, and healthy stdout tool progress.
  • New cases are in the existing portable tests/test_run_exec_stream_callback.py module.
  • Ruff and git diff --check passed. Remote CI remains to be checked.

Copilot AI lite review requested due to automatic review settings September 7, 2026 14:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new check_idle_deadlines() helper can continue into escalation/callback logic after a termination condition is met, which can cause duplicate termination/emits and overwrite the intended watchdog reason.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR addresses a runner watchdog correctness issue where continuous stderr diagnostics could keep the stream “active” and prevent idle watchdog checks from firing, allowing stalled stdout/model progress to go undetected indefinitely. The changes keep the stdout activity clock independent from stderr, while still performing idle deadline checks even when stderr keeps the queue nonempty.

Changes:

  • Refactors idle watchdog evaluation into a helper and runs it on stderr processing as well as on empty-queue timeouts.
  • Ensures only stdout activity resets the idle escalation state.
  • Adds regression tests that simulate high-volume stderr chatter and validate warning/stall/terminate behavior.
File summaries
File Description
argus_skill/agent_cli/_run_exec.py Adjusts the streaming loop so stderr diagnostics don’t reset the idle clock and so idle checks still execute during stderr-heavy output.
tests/test_run_exec_stream_callback.py Adds real-subprocess regression tests covering stderr chatter, hard idle deadlines, stdout progress, and inactivity callbacks.
Review details

Suppressed comments (1)

argus_skill/agent_cli/_run_exec.py:459

  • After inactivity_callback returns "restart", the process is terminated and state.watchdog_terminated is set, but the function continues into idle escalation stages. This can emit additional watchdog warnings/termination and potentially override the restart reason; exit early once restart is requested.
                    self._terminate_process(
                        process,
                        include_detached_children=self.backend == BACKEND_OPENCODE,
                    )
                    state.watchdog_terminated = True
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +424 to +426
check_external_interrupt()
check_wall_clock_limit()

@lbx154 lbx154 closed this Sep 14, 2026
@lbx154 lbx154 reopened this Sep 14, 2026
@lbx154
lbx154 changed the base branch from main to dev September 14, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants