Skip to content

🐛 fix(kick-logs): archive kick log when the turn ends, not only at the next rotation point - #15270

Merged
clubanderson merged 2 commits into
v5from
fix/kick-log-archive-turn-end
Oct 29, 2026
Merged

clubanderson merged 2 commits into
v5from
fix/kick-log-archive-turn-end

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Problem

A kick's log is archived to /data/logs/kicks/<agent>/ only at a rotation point — the next kick (rotateKickLogOnKickLocked), an agent restart (tearDownTurnLocked) or graceful shutdown (ArchiveAllKickLogs, SIGTERM). A turn that simply ends is never archived; its only copy is the tmux scrollback until something else happens. For a daily-cadence agent that is a ~24 h window, and a hard container death inside it (OOM kill, non-zero exit, node loss — no SIGTERM) takes the tmux server and the run with it. 🕘 past kicks then shows nothing for that day.

Evidence (hosted hive hosted-available-vllmd-260731-fikn, hive 5.186.4-673)

  • 2026-10-28T07:01:31-04:00 — audit: agent kicked (scanner)
  • +22 s — kick turn ended (outcome ended); again at 13:11:33 after the hive restarted (seconds_since_kick 22202)
  • 17:08:47Z — container exited exit code 17 ("Error"), restarted 17:09:50Z. Not a graceful SIGTERM, so ArchiveAllKickLogs("shutdown") never ran.
  • /data/logs/kicks/scanner/ newest archive: 20261016-110450.534-shutdown.log. The Oct 28 run's output no longer exists anywhere.

Reported by Mario Alvarado (IBM) via Slack — hive hosted-available-vllmd-260731-fikn, scanner run 2026-10-28 05:01 America/Mexico_City whose log never appeared under past kicks.

Fixes #15264

Fix

  • Archive on turn end. settleKickOutcomeLocked (the event that logs kick turn ended / settles KickOutcome) now calls archiveKickLogOnTurnEndLocked, which archives under reason turn-end when kickLogPending. It does not clear the scrollback or kickLogPending: the pane keeps showing the result, and the next rotation point still takes the final snapshot and clears, exactly as today.
  • One archive per kick, latest content wins. AgentProcess tracks kickLogArchivedKick/kickLogArchivedPath. When a later archive is written for the same LastKick (same timestamp stem, different reason suffix) the superseded file is removed before pruning — so a turn-end file is replaced by the kick/restart/shutdown snapshot of the same run; a second turn-end for the same kick overwrites the same filename. Retention therefore still counts kicks, not files, and the newest file is never pruned.
  • Turn-loss accounting. tearDownTurnLocked still archives a settled turn (final content) but no longer records it as an interrupted turn or arms the restart/kick breaker — a turn that was seen to end was not lost.
  • Dashboard. History page copy ("No archived kick logs yet…", live-log blurb) says archives are written when a kick's turn ends; trigger column gets a tooltip explaining turn-end / kick / restart / shutdown; past kicks button tooltips in index.html updated.
  • Linear responder comment updated: kick-log-archived now fires at turn end (more accurate "run finished"); the tracker already deactivates the agent mapping on finish, so the later rotation archive does not double-notify.
  • Docs: docs/agent-logging.md rotation-point section and limitations; changelog.d/fixed-kick-log-archive-on-turn-end.md.

Tests (src/crux/agent/kick_logs_turn_end_test.go, no time.Sleep)

  • settle → one <LastKick>-turn-end.log, scrollback not cleared, kickLogPending still set
  • nothing pending → no archive
  • second turn-end for the same kick overwrites, no duplicate
  • next-kick rotation after a turn-end archive → one -kick.log for that kick with final content, scrollback cleared once
  • distinct kicks keep distinct archives; retention 2 across supersedes keeps 2 kicks and never prunes the newest file
  • teardown of a settled turn → archived (shutdown) but TurnLoss unchanged, (false,false)
  • teardown of an unsettled turn → unchanged crash path (interrupted, bytes counted)
  • superseded file already gone → non-fatal
  • pkg/dashboard: history page asserts the new copy

Local: go test -coverprofile ./crux/agent/ 89.0% (only failure is the pre-existing macOS tmux socket-path-too-long env issue in tmux_inherited_env_test.go); ./pkg/dashboard/ ok, 87.9%.

@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15253 🐛 decision toast: issue decisions no longer read as 'approved for merge' same files src/pkg/dashboard/static/index.html
#15219 🐛 fix: preserve human and owner PRs during delegated decisions same files src/pkg/dashboard/static/index.html
#15279 🐛 dashboard: merged toast shows the PR number same files src/pkg/dashboard/static/index.html
#15186 🐛 Activity Monitor: one coherent post-decision state for Let hive decide rows; 060 reviewed in CI / Review same files src/pkg/dashboard/static/index.html
#15281 ✨ tracker: rework pickup signal on the waiting-on chip same files src/pkg/dashboard/static/index.html
#15134 🐛 Delegated decision on a capability gap: model the blocker, stop the un-park/re-park loop, name the missing App permission same files src/pkg/dashboard/static/index.html
#15104 🐛 Activity Monitor: Rebase, then merge as maintainer; fixer owns parked conflicting PRs; dismiss self-inflicted changes_requested same files src/pkg/dashboard/static/index.html

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 29, 2026
…e next rotation point

A kick's log was archived only at a rotation point (next kick, restart,
graceful shutdown). A turn that ended lived only in tmux scrollback until
then; a hard container death in that window (exit 17, OOM — no SIGTERM)
lost the run, and past kicks showed nothing for the day.

Archive under reason turn-end when settleKickOutcomeLocked observes the
turn end. Scrollback and kickLogPending are left alone so the next
rotation point still takes the final snapshot and clears; that snapshot
supersedes the turn-end file (same LastKick stem) so each kick keeps one
archive and retention still counts kicks. A settled turn torn down later
is archived but no longer recorded as an interrupted turn.

Fixes #15264

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the fix/kick-log-archive-turn-end branch from b07cb3c to 2ca7ba6 Compare October 29, 2026 03:05
@kubestellar-prow kubestellar-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot added role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 29, 2026
Signed-off-by: Andy Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One concrete blocker, otherwise the logic reads correct to me.

Blocker — changelog fragment is in the wrong directory. The fragment was added at src/changelog.d/fixed-kick-log-archive-on-turn-end.md; the guard requires repo-root changelog.d/ and changelog-fragment-guard is failing on this head for exactly that reason (run 40810350468: fragment is under src/changelog.d/; fragments live in changelog.d/ at the repo root). src/changelog.d/README.md says the same. The PR body cites changelog.d/fixed-kick-log-archive-on-turn-end.md, so this is a placement slip, not a design choice. Fix:

git mv src/changelog.d/fixed-kick-log-archive-on-turn-end.md changelog.d/fixed-kick-log-archive-on-turn-end.md

What I verified in the Go change:

  • Supersede matching relies on filenameTime being the previous kick's LastKick when the kick rotation archives. rotateKickLogOnKickLocked runs at manager_kick.go:309 and agent.LastKick = &now is set at :447, so the turn-end and kick files share a stem and the earlier one is removed (kick_logs.go:231). Distinct kicks have distinct stems and both files survive.
  • settleKickOutcomeLocked is called under m.mu in both callers (manager_kick.go:1040, kick_outcome.go:230), and archiveKickLogBytesLocked already captured scrollback under the lock from the rotation path, so the new call site adds no new locking shape.
  • tearDownTurnLocked (turn_loss.go:110): archiving still happens before the Settled early return, so a restart/shutdown after a settled turn still takes the final snapshot; only the interruption accounting and breaker arming are skipped. manager_restart.go:707 reads producing from this — it now stays false for a settled turn, which is the intent. Both branches are covered by TestTearDown_SettledTurnIsArchivedButNotInterrupted / TestTearDown_UnsettledTurnStillCountsAsInterrupted.
  • kick-log-archived now fires twice per kick; the only consumers are pkg/linearagent/responder.go:158 (drops the mapping after the first, as the body says) and the sandbox path, which emits its own reason.
  • go build ./crux/agent/ ./pkg/dashboard/ passes on the head.

Minor: the body still carries <!-- TODO(owner): @-mention … -->; harmless, but worth resolving or removing before merge.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm requested changes

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@clubanderson

Copy link
Copy Markdown
Member Author

/lgtm
/approve

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@clubanderson: you cannot LGTM your own PR.

Details

In response to this:

/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 29, 2026
@clubanderson
clubanderson merged commit e6b90a3 into v5 Oct 29, 2026
81 of 83 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/kick-log-archive-turn-end branch October 29, 2026 03:56
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

hivecommons-hive Bot added a commit that referenced this pull request Oct 29, 2026
* chore(crux): move tmux session accessors out of manager.go

SessionMissing and TmuxSession are tmux-session queries; they belong with
the rest of the tmux boundary in manager_tmux.go. #15270 grew
agent/manager.go to 1696 lines, over the 1689-line baseline enforced by
crux's Go file-size ratchet, so every crux-publish sync since then fails
that check on crux main. Pure move, no behaviour change.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* changelog: fragment for #15330

Generated by changelog-fragment-guard from the PR title (#5675).
Edit the wording freely, or replace this file with the no-changelog
label if the change is not user-visible.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>

---------

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/L Denotes a PR that changes 100-499 lines, ignoring generated files. stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 kick log is lost when the container crashes after the turn ends but before the next kick/restart/shutdown

1 participant