Skip to content

πŸ› watchdog: stop paging PaneShowsLogin on a relaunched CLI's login flash - #17323

Merged
clubanderson merged 3 commits into
v5from
fix/watchdog-login-flash-relaunch
Nov 9, 2026
Merged

clubanderson merged 3 commits into
v5from
fix/watchdog-login-flash-relaunch

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #17320

Problem

2026-11-09 00:50Z on a hosted spoke, three copilot agents (reviewer, adjudicator, ci-maintainer) were paged needs re-authentication (PaneShowsLogin) for one tick, then cleared. Their Copilot CLIs had all just been relaunched; the CLI paints login chrome for ~0.5 s (Login status unknown) before self-fetch succeeds. Classify() suppressed login chrome only inside BootGrace of StartedAt β€” the agent's original start, 22 h earlier β€” so a mid-life relaunch got zero grace and one sighting paged.

Fix

  1. Grace follows the latest CLI launch. Observation.LastLaunchAt is populated from a new AgentProcess.lastCLILaunchAt, recorded by markCLILaunchedLocked at the point launchInTmux types the CLI into the pane. Classify dates boot grace from max(StartedAt, LastLaunchAt).
  2. Multi-poll streak before paging. ApplyLoginStreak (new login_streak.go, applied by Reconciler.classifyLocked) downgrades auth-required to unknown with reason login chrome seen N/M polls until login chrome has been seen on governor.watchdog.liveness.login_streak consecutive polls (default 3, mirroring the pane poller's loginStreakRestartMin). Any other verdict resets the streak. login_streak: 1 disables the guard. A pane genuinely stuck at login still pages once the streak is met.

reconciler.go and manager_launch.go are at their crux file-size baselines, so the new logic lives in login_streak.go / watchdog_fleet.go and reconciler.go is net-zero lines.

Tests

  • TestClassifyLoginChromeAfterMidLifeRelaunch: relaunch inside grace β†’ Unknown (incl. missing session); past grace / no relaunch / stale LastLaunchAt β†’ AuthRequired.
  • TestApplyLoginStreak: 1/3, 2/3 β†’ Unknown with the streak reason; 3/3 and beyond β†’ AuthRequired; required ≀ 1 disables; non-auth verdicts pass through.
  • TestAuthLoginStreak (reconciler, default streak): one-poll flash never pages and the streak resets when chrome leaves; chrome on every poll pages exactly at the threshold, no restart.
  • Settings tests cover login_streak explicit/negative handling. fastSettings() sets LoginStreak = 1 so existing single-tick auth tests keep asserting the paging branch.

Docs: src/docs/agent-watchdog.md auth-probing section and config example.

A Copilot CLI relaunched mid-life (kick restart, memory-drain recycle,
watchdog restart) paints login chrome for ~0.5s while it re-validates its
token. Classify dated boot grace from the agent's original StartedAt, so a
22h-old agent got no grace and a single sighting flipped
Authenticated=False/PaneShowsLogin and paged the operator.

- Observation.LastLaunchAt records the most recent CLI spawn
  (markCLILaunchedLocked); boot grace uses the later of StartedAt and it.
- ApplyLoginStreak holds auth-required back until login chrome has been
  seen on Settings.LoginStreak consecutive polls (liveness.login_streak,
  default 3), mirroring the pane poller's loginStreak guard. Below the
  threshold the pane is unknown with reason 'login chrome seen 1/3 polls'.

Fixes #17320

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 9, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

/lgtm
/approve

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@clubanderson: you cannot LGTM your own PR.

Details

In response to this:

/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 9, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17326 πŸ› dashboard: widen the Recent merges popover so titles are readable same files src/pkg/dashboard/static/index.html
#17325 πŸ› Activity Monitor: secondary issue of a shared PR rides the PR's lane same files src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • coverage

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency (read at head 4a45700)

src/docs/agent-watchdog.md and the changelog fragment match the implemented behavior (Ready=Unknown / login chrome seen N/M polls confirmed in reconciler.go:545-556 + classify.go:ApplyLoginStreak). Two other operator-facing surfaces that enumerate the governor.watchdog.liveness.* keys were not updated for the new login_streak knob:

  1. src/hive.yaml.example:479-481 β€” the commented watchdog liveness block lists stuck_overlay_after / shell_prompt_after only.

      #   liveness:
      #     stuck_overlay_after: 10m          # Unanswered modal older than this -> restart
      #     shell_prompt_after: 5m            # Bare shell / silent pane grace before restart
      #     login_streak: 3                   # Consecutive polls of login chrome before auth-required pages
    
  2. src/pkg/dashboard/static-src/js/config-governor.js:1783 β€” the Agent Watchdog panel's "YAML keys" help text lists every liveness.* key except the new one; add <code>governor.watchdog.liveness.login_streak</code> after liveness.stuck_overlay_after.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

…atchdog settings help

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 4a45700) β€” approve. Lock discipline is right (classifyLocked runs under r.mu, lastCLILaunchAt is written under m.mu in launchInTmux and read under RLock in WatchdogFleet.Observe), reconcileAuth keys off cls.Class so the streak guard is honoured, and the only path that types a CLI into the pane now goes through markCLILaunchedLocked (the "CLI already running" branch at manager_launch.go:218 correctly leaves it alone). One advisory:

Low β€” streak window can report Authenticated=True/CredentialPresent over visible login chrome. ApplyLoginStreak downgrades polls 1..N-1 to ClassUnknown (classify.go:327-330). In reconcileAuth that means healable is false and neither the PaneShowsLogin branch nor the LoginPromptWithUsableCredential branch fires (reconciler.go:731-752), so the fall-through upgrade at reconciler.go:788-791 sets Authenticated=True, CredentialPresent whenever AuthKnown && AuthAvailable β€” exactly the case the comment above it says must not happen "on an agent visibly sitting at a login prompt". For a genuinely logged-out pane that is a false True for up to two polls (β‰ˆ10 min at the default 300 s probe interval) before it flips to False. Boot grace had the same hole before this PR, so this widens it rather than introduces it; non-blocking, but a cheap fix:

	if status == ConditionUnknown && !healable && !obs.ShowsLoginPrompt && obs.AuthKnown && obs.AuthAvailable {

(at reconciler.go:788). Tests look adequate for the new paths; CI build-go was still pending when I read this.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit eb6c66a into v5 Nov 9, 2026
84 of 85 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/watchdog-login-flash-relaunch branch November 9, 2026 01:30
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

watchdog: PaneShowsLogin pages on Copilot CLI relaunch login flash

1 participant