Skip to content

πŸ› fix: stop filing the pre-restart kick-undeliverable warning as a fleet issue - #15315

Merged
hivecommons-hive[bot] merged 1 commit into
hivecommons:v5from
odttlnt:fix/agent-pane-timeout-kick-15313
Oct 29, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
hivecommons:v5from
odttlnt:fix/agent-pane-timeout-kick-15313

Conversation

@odttlnt

@odttlnt odttlnt commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Fixes #15313

crux/agent/busy_visibility.go logs kick undeliverable; busy-no-activity restart not yet attempted, not self-reporting at WARN on every refused kick until the stall restart has run (decision B on #12787: that condition is a dashboard warning, not a fleet issue). The slog feed still counted the WARN, so 33 repeats in 10m crossed the threshold and filed #15313 (agent_pane_timeout).

This adds the message to benignSubstrings in pkg/fleetreport/fleetreport.go, so the generic feed drops it. The WARN stays in local logs. Kicks that keep failing after the restart still report via kick_failed_repeatedly (different message), and other kick-undeliverable warnings remain reportable.

Tests: TestIsBenignMessageKickUndeliverableBeforeRestart (positive case plus the post-restart report and other kick warnings as negatives). Not run locally per AGENTS.md (CI is the verdict; no Go toolchain here).

β€” hive: backend=bob model=premium

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@odttlnt this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • build-and-test
  • coverage
  • golangci-lint
  • test
  • test (dashboard 1/4)
  • test (dashboard 2/4)
  • test (dashboard 3/4)
  • test (dashboard 4/4)
  • test (dashboard shuffle 1/3)
  • test (dashboard shuffle 2/3)
  • test (dashboard shuffle 3/3)

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the source of the message and the matcher; this looks correct to me.

  • The new benign fragment is a verbatim substring of the WARN at src/crux/agent/busy_visibility.go:178 ("kick undeliverable; busy-no-activity restart not yet attempted, not self-reporting"), and IsBenignMessage lower-cases the input before strings.Contains (fleetreport.go, m := strings.ToLower(msg) β†’ loop at L864), so the already-lower-case fragment matches.
  • The post-restart path still reports: busy_visibility.go:187 and :225 observe ClassKickFailedRepeatedly with a different message, and the new test's negative cases cover that message plus the two other kick undeliverable WARNs, so neither becomes benign by accident.
  • The sibling INFO at busy_visibility.go:161 ("kick undeliverable while agent turn in progress; not self-reporting") is not matched by this fragment; that is fine since INFO lines are not fed to the defect filer, but worth knowing if it ever gets promoted to WARN.

No findings.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Shared CI baseline incident β€” not caused by this PR. The failing pkg/dashboard shards (and the downstream test, coverage, build-and-test, golangci-lint checks) are red on v5 HEAD itself:

pkg/dashboard/stage_tracker_linked_progress_test.go:95:31: not enough arguments in call to stageTrackerResolveLinkedPRs

Tracked in #15322; fix in #15321. No action needed here β€” once #15321 lands, this PR will go green on its next run / rebase.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15374 πŸ› fix: suppress gateway errors in log messages same files src/pkg/fleetreport/fleetreport.go

…eet issue

The kick-undeliverable WARN emitted before busy-no-activity restart has
been attempted states that it is not self-reporting (decision B on hivecommons#12787),
yet the generic repeated-error feed counted it and filed an
agent_pane_timeout issue. Mark the message benign for that feed; kicks that
still fail after the restart keep reporting as kick_failed_repeatedly.

Hive-Run: hivecommons#15313
Hive-Plan: selfreport agent_pane_timeout, pre-restart kick-undeliverable warning
Hive-Spec: fleetreport-benign-messages#kick-undeliverable-pre-restart
Signed-off-by: SACHIN NAIK <sachin.naik@ibm.com>
@odttlnt
odttlnt force-pushed the fix/agent-pane-timeout-kick-15313 branch from 48a99dc to 97a705c Compare October 29, 2026 05:52

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me.

Verified against the tree:

  • src/crux/agent/busy_visibility.go:178 emits exactly kick undeliverable; busy-no-activity restart not yet attempted, not self-reporting β€” the new benignSubstrings entry in fleetreport.go is a verbatim, already-lower-case suffix of that line, and IsBenignMessage matches via strings.Contains on the lower-cased message, so the WARN is dropped by the feed.
  • The post-restart report (…after busy-no-activity restart did not recover the session…) does not contain not yet attempted, so it stays reportable, as the new test's negative cases assert.
  • isBenign on the ACMM path (fleetreport.go:696) only excludes I/O timeouts and does not consult benignSubstrings, so this change is scoped to the repeated-error feed, matching the body's claim.

Changelog fragment name follows the fixed-<issue>-…md convention.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Ready for a human merge: verdict approved @97a705c, 54/54 checks green, DCO yes, soak 30m0s elapsed. Blocked only by role/newcomer (below require_tiers).

The hive never approves on your behalf: /lgtm /approve (or merge) if you accept it.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test coverage (quality) β€” approve, no findings (read at head 97a705c).

TestIsBenignMessageKickUndeliverableBeforeRestart fails without the new benignSubstrings entry and pins both arms: the pre-restart message is benign, while the post-restart kick undeliverable N times in a row after busy-no-activity restart… report, the TUI-menu variant and the re-engagement variant stay reportable β€” so the substring chosen cannot silently widen to swallow kick_failed_repeatedly. Nothing left to pin.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @clubanderson for Hive auto-merge on green CI.

Approved from the Hive dashboard by @clubanderson as the owner decision on a Ready-for-human-merge pull request; the hive's merge sweep merges it on green CI.

@hivecommons-hive hivecommons-hive Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

@hivecommons-hive[bot]: changing LGTM is restricted to collaborators

Details

In response to this:

Approved by @clubanderson for Hive auto-merge on green CI.

Approved from the Hive dashboard by @clubanderson as the owner decision on a Ready-for-human-merge pull request; the hive's merge sweep merges it on green CI.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive queued this PR but could not set the prow/lgtm commit status, so Prow may strip the lgtm label and the auto-merge queue may not hold.

@hivecommons-hive
hivecommons-hive Bot merged commit 687c3cb into hivecommons:v5 Oct 29, 2026
63 of 64 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#1531597a705c` sha256:2b8b7157c9be3e06d2357898d94d609b042835f1a75523cf9fc9086245de7c89 β€” download from the dashboard or `hivectl review evidence #15315`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merged by hive contributor-lane auto-merge: CI green for β‰₯30m, reviewer verdict approve, @clubanderson approved this head as the owner decision (admitted_by: operator-approval:clubanderson) β€” merged as-is on that approval.

hivecommons-hive Bot added a commit that referenced this pull request Oct 29, 2026
…87c3cb (#15488)

The post-merge trailer check failed v5 687c3cb (squash of #15315):
GitHub wrote the PR author's private-email identity
91499691+odttlnt@users.noreply.github.com while the branch's
Signed-off-by stayed sachin.naik@ibm.com. Same person; pre-merge DCO
passed because the commits API resolves that address to @odttlnt.

- check-dco-trailers.sh: same_person_noreply_evidence accepts a noreply
  author signed off under another address when the inspected branch's
  own history (DCO_SAME_PERSON_WINDOW, default 500) shows the link:
  authored-under-signoff-email (same name authored/committed under the
  sign-off address) or login-and-email-cooccur (both identities on one
  commit). No GitHub API call. Reported as an informational
  same-person-noreply line with the evidence commit, counted in the
  summary; a waiver on an accepted commit is STALE-WAIVER as before.
  DCO_SAME_PERSON_EVIDENCE_REFS adds refs for local checks.
- lib-dco-identity.sh: github_noreply_login helper (pure).
- Waive 687c3cb: v5 has no prior commit
  under the work address (only the PR branch commit 97a705c), so the
  heuristic has nothing to cite yet.
- Self-tests for both evidence kinds, the three negatives (unknown
  address, different name, repeated ambiguous shape), window=0, waiver
  fallback, stale-waiver, evidence refs and config errors.
- CONTRIBUTING.md / v5-sync-policy.md document the rule.

Fixes #15401

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[selfreport] agent_pane_timeout: kick undeliverable; busy-no-activity restart not yet attempted, not self-report…

1 participant