Repository navigation
🐛 fix(kick-logs): archive kick log when the turn ends, not only at the next rotation point - #15270
Conversation
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
…e next rotation point A kick's log was archived only at a rotation point (next kick, restart, graceful shutdown). A turn that ended lived only in tmux scrollback until then; a hard container death in that window (exit 17, OOM — no SIGTERM) lost the run, and past kicks showed nothing for the day. Archive under reason turn-end when settleKickOutcomeLocked observes the turn end. Scrollback and kickLogPending are left alone so the next rotation point still takes the final snapshot and clears; that snapshot supersedes the turn-end file (same LastKick stem) so each kick keeps one archive and retention still counts kicks. A settled turn torn down later is archived but no longer recorded as an interrupted turn. Fixes #15264 Signed-off-by: Andy Anderson <andy@clubanderson.com>
b07cb3c to
2ca7ba6
Compare
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
Signed-off-by: Andy Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
One concrete blocker, otherwise the logic reads correct to me.
Blocker — changelog fragment is in the wrong directory. The fragment was added at src/changelog.d/fixed-kick-log-archive-on-turn-end.md; the guard requires repo-root changelog.d/ and changelog-fragment-guard is failing on this head for exactly that reason (run 40810350468: fragment is under src/changelog.d/; fragments live in changelog.d/ at the repo root). src/changelog.d/README.md says the same. The PR body cites changelog.d/fixed-kick-log-archive-on-turn-end.md, so this is a placement slip, not a design choice. Fix:
git mv src/changelog.d/fixed-kick-log-archive-on-turn-end.md changelog.d/fixed-kick-log-archive-on-turn-end.mdWhat I verified in the Go change:
- Supersede matching relies on
filenameTimebeing the previous kick'sLastKickwhen thekickrotation archives.rotateKickLogOnKickLockedruns atmanager_kick.go:309andagent.LastKick = &nowis set at:447, so theturn-endandkickfiles share a stem and the earlier one is removed (kick_logs.go:231). Distinct kicks have distinct stems and both files survive. settleKickOutcomeLockedis called underm.muin both callers (manager_kick.go:1040,kick_outcome.go:230), andarchiveKickLogBytesLockedalready captured scrollback under the lock from the rotation path, so the new call site adds no new locking shape.tearDownTurnLocked(turn_loss.go:110): archiving still happens before theSettledearly return, so a restart/shutdown after a settled turn still takes the final snapshot; only the interruption accounting and breaker arming are skipped.manager_restart.go:707readsproducingfrom this — it now stays false for a settled turn, which is the intent. Both branches are covered byTestTearDown_SettledTurnIsArchivedButNotInterrupted/TestTearDown_UnsettledTurnStillCountsAsInterrupted.kick-log-archivednow fires twice per kick; the only consumers arepkg/linearagent/responder.go:158(drops the mapping after the first, as the body says) and the sandbox path, which emits its own reason.go build ./crux/agent/ ./pkg/dashboard/passes on the head.
Minor: the body still carries <!-- TODO(owner): @-mention … -->; harmless, but worth resolving or removing before merge.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
/lgtm |
|
@clubanderson: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
* chore(crux): move tmux session accessors out of manager.go SessionMissing and TmuxSession are tmux-session queries; they belong with the rest of the tmux boundary in manager_tmux.go. #15270 grew agent/manager.go to 1696 lines, over the 1689-line baseline enforced by crux's Go file-size ratchet, so every crux-publish sync since then fails that check on crux main. Pure move, no behaviour change. Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> * changelog: fragment for #15330 Generated by changelog-fragment-guard from the PR title (#5675). Edit the wording freely, or replace this file with the no-changelog label if the change is not user-visible. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com> --------- Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Problem
A kick's log is archived to
/data/logs/kicks/<agent>/only at a rotation point — the next kick (rotateKickLogOnKickLocked), an agent restart (tearDownTurnLocked) or graceful shutdown (ArchiveAllKickLogs, SIGTERM). A turn that simply ends is never archived; its only copy is the tmux scrollback until something else happens. For a daily-cadence agent that is a ~24 h window, and a hard container death inside it (OOM kill, non-zero exit, node loss — no SIGTERM) takes the tmux server and the run with it. 🕘 past kicks then shows nothing for that day.Evidence (hosted hive
hosted-available-vllmd-260731-fikn, hive 5.186.4-673)2026-10-28T07:01:31-04:00—audit: agent kicked(scanner)kick turn ended(outcomeended); again at 13:11:33 after the hive restarted (seconds_since_kick 22202)17:08:47Z— container exited exit code 17 ("Error"), restarted17:09:50Z. Not a graceful SIGTERM, soArchiveAllKickLogs("shutdown")never ran./data/logs/kicks/scanner/newest archive:20261016-110450.534-shutdown.log. The Oct 28 run's output no longer exists anywhere.Reported by Mario Alvarado (IBM) via Slack — hive
hosted-available-vllmd-260731-fikn, scanner run 2026-10-28 05:01 America/Mexico_City whose log never appeared under past kicks.Fixes #15264
Fix
settleKickOutcomeLocked(the event that logskick turn ended/ settlesKickOutcome) now callsarchiveKickLogOnTurnEndLocked, which archives under reasonturn-endwhenkickLogPending. It does not clear the scrollback orkickLogPending: the pane keeps showing the result, and the next rotation point still takes the final snapshot and clears, exactly as today.AgentProcesstrackskickLogArchivedKick/kickLogArchivedPath. When a later archive is written for the sameLastKick(same timestamp stem, different reason suffix) the superseded file is removed before pruning — so aturn-endfile is replaced by thekick/restart/shutdownsnapshot of the same run; a second turn-end for the same kick overwrites the same filename. Retention therefore still counts kicks, not files, and the newest file is never pruned.tearDownTurnLockedstill archives a settled turn (final content) but no longer records it as an interrupted turn or arms the restart/kick breaker — a turn that was seen to end was not lost.triggercolumn gets a tooltip explainingturn-end/kick/restart/shutdown;past kicksbutton tooltips inindex.htmlupdated.kick-log-archivednow fires at turn end (more accurate "run finished"); the tracker already deactivates the agent mapping on finish, so the later rotation archive does not double-notify.docs/agent-logging.mdrotation-point section and limitations;changelog.d/fixed-kick-log-archive-on-turn-end.md.Tests (
src/crux/agent/kick_logs_turn_end_test.go, notime.Sleep)<LastKick>-turn-end.log, scrollback not cleared,kickLogPendingstill set-kick.logfor that kick with final content, scrollback cleared onceshutdown) butTurnLossunchanged,(false,false)pkg/dashboard: history page asserts the new copyLocal:
go test -coverprofile ./crux/agent/89.0% (only failure is the pre-existing macOS tmux socket-path-too-long env issue intmux_inherited_env_test.go);./pkg/dashboard/ok, 87.9%.