Repository navigation
π watchdog: stop paging PaneShowsLogin on a relaunched CLI's login flash - #17323
Conversation
A Copilot CLI relaunched mid-life (kick restart, memory-drain recycle, watchdog restart) paints login chrome for ~0.5s while it re-validates its token. Classify dated boot grace from the agent's original StartedAt, so a 22h-old agent got no grace and a single sighting flipped Authenticated=False/PaneShowsLogin and paged the operator. - Observation.LastLaunchAt records the most recent CLI spawn (markCLILaunchedLocked); boot grace uses the later of StartedAt and it. - ApplyLoginStreak holds auth-required back until login chrome has been seen on Settings.LoginStreak consecutive polls (liveness.login_streak, default 3), mirroring the pane poller's loginStreak guard. Below the threshold the pane is unknown with reason 'login chrome seen 1/3 polls'. Fixes #17320 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
/lgtm |
|
@clubanderson: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. @clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
docs-currency (read at head 4a45700)
src/docs/agent-watchdog.md and the changelog fragment match the implemented behavior (Ready=Unknown / login chrome seen N/M polls confirmed in reconciler.go:545-556 + classify.go:ApplyLoginStreak). Two other operator-facing surfaces that enumerate the governor.watchdog.liveness.* keys were not updated for the new login_streak knob:
-
src/hive.yaml.example:479-481β the commented watchdog liveness block listsstuck_overlay_after/shell_prompt_afteronly.# liveness: # stuck_overlay_after: 10m # Unanswered modal older than this -> restart # shell_prompt_after: 5m # Bare shell / silent pane grace before restart # login_streak: 3 # Consecutive polls of login chrome before auth-required pages -
src/pkg/dashboard/static-src/js/config-governor.js:1783β the Agent Watchdog panel's "YAML keys" help text lists everyliveness.*key except the new one; add<code>governor.watchdog.liveness.login_streak</code>afterliveness.stuck_overlay_after.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
β¦atchdog settings help Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Correctness review (head 4a45700) β approve. Lock discipline is right (classifyLocked runs under r.mu, lastCLILaunchAt is written under m.mu in launchInTmux and read under RLock in WatchdogFleet.Observe), reconcileAuth keys off cls.Class so the streak guard is honoured, and the only path that types a CLI into the pane now goes through markCLILaunchedLocked (the "CLI already running" branch at manager_launch.go:218 correctly leaves it alone). One advisory:
Low β streak window can report Authenticated=True/CredentialPresent over visible login chrome. ApplyLoginStreak downgrades polls 1..N-1 to ClassUnknown (classify.go:327-330). In reconcileAuth that means healable is false and neither the PaneShowsLogin branch nor the LoginPromptWithUsableCredential branch fires (reconciler.go:731-752), so the fall-through upgrade at reconciler.go:788-791 sets Authenticated=True, CredentialPresent whenever AuthKnown && AuthAvailable β exactly the case the comment above it says must not happen "on an agent visibly sitting at a login prompt". For a genuinely logged-out pane that is a false True for up to two polls (β10 min at the default 300 s probe interval) before it flips to False. Boot grace had the same hole before this PR, so this widens it rather than introduces it; non-blocking, but a cheap fix:
if status == ConditionUnknown && !healable && !obs.ShowsLoginPrompt && obs.AuthKnown && obs.AuthAvailable {
(at reconciler.go:788). Tests look adequate for the new paths; CI build-go was still pending when I read this.
β hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #17320
Problem
2026-11-09 00:50Z on a hosted spoke, three copilot agents (reviewer, adjudicator, ci-maintainer) were paged
needs re-authentication (PaneShowsLogin)for one tick, then cleared. Their Copilot CLIs had all just been relaunched; the CLI paints login chrome for ~0.5 s (Login status unknown) beforeself-fetchsucceeds.Classify()suppressed login chrome only insideBootGraceofStartedAtβ the agent's original start, 22 h earlier β so a mid-life relaunch got zero grace and one sighting paged.Fix
Observation.LastLaunchAtis populated from a newAgentProcess.lastCLILaunchAt, recorded bymarkCLILaunchedLockedat the pointlaunchInTmuxtypes the CLI into the pane.Classifydates boot grace frommax(StartedAt, LastLaunchAt).ApplyLoginStreak(newlogin_streak.go, applied byReconciler.classifyLocked) downgradesauth-requiredtounknownwith reasonlogin chrome seen N/M pollsuntil login chrome has been seen ongovernor.watchdog.liveness.login_streakconsecutive polls (default 3, mirroring the pane poller'sloginStreakRestartMin). Any other verdict resets the streak.login_streak: 1disables the guard. A pane genuinely stuck at login still pages once the streak is met.reconciler.goandmanager_launch.goare at their crux file-size baselines, so the new logic lives inlogin_streak.go/watchdog_fleet.goandreconciler.gois net-zero lines.Tests
TestClassifyLoginChromeAfterMidLifeRelaunch: relaunch inside grace β Unknown (incl. missing session); past grace / no relaunch / stale LastLaunchAt β AuthRequired.TestApplyLoginStreak: 1/3, 2/3 β Unknown with the streak reason; 3/3 and beyond β AuthRequired;required β€ 1disables; non-auth verdicts pass through.TestAuthLoginStreak(reconciler, default streak): one-poll flash never pages and the streak resets when chrome leaves; chrome on every poll pages exactly at the threshold, no restart.login_streakexplicit/negative handling.fastSettings()setsLoginStreak = 1so existing single-tick auth tests keep asserting the paging branch.Docs:
src/docs/agent-watchdog.mdauth-probing section and config example.