Repository navigation
fix: stop dropping tail_500 from pane-timeout self-report diagnostics - #15319
hivecommons-hive[bot] merged 2 commits into
Conversation
The slogfeed WARN/ERROR tee that feeds fleet self-reports bounds the attribute excerpt it attaches to an observation at 512 bytes, writing attrs in record order and stopping at the first one that would cross the cap. agent.manager's "pane content at timeout" WARN logs head_500 before tail_500, each up to 500 runes (worse with multi-byte box-drawing characters like ❯ and ╰─); head_500 alone routinely fills 512 bytes, so tail_500 — the pane's CURRENT state, and the only field useful for diagnosing a stuck CLI — was dropped every single time. Both hivecommons#14586 and hivecommons#15311 self-reported this exact WARN with no usable excerpt attached, and hivecommons#14586 was closed uninvestigated for lack of one. Raise the cap to 4096 bytes, comfortably fitting both excerpts plus the preceding flags, well under the tracker's own 8KiB storage bound. Closes hivecommons#15311 Hive-Run: hivecommons#15311 Hive-Plan: fix pane-timeout selfreport diagnostics truncation Hive-Spec: selfreport-diagnostics#agent_pane_timeout Signed-off-by: onkar717 <shelkeonka@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@onkar717 this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Your relay session received this as a |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
…nt-pane-timeout-content Signed-off-by: onkar717 <shelkeonka@gmail.com>
There was a problem hiding this comment.
Looks correct to me. The mechanism described is real and the fix is the minimal one.
What I verified at ebe3d72, reading the tree rather than the diff:
src/pkg/fleetreport/slogfeed/slogfeed.go:181-208—diagnostics()writes attrs in record order andwrite()refuses onceb.Len() >= maxDiagnosticsBytes(L186), then hard-truncates at L205. So an attr that fills the budget drops every attr after it, not just the overflow.src/crux/agent/manager.go:1275-1284— the WARN logshead_500beforetail_500, eachtruncate*(output, 500)runes. 500 × 3-byte box-drawing runes = 1500 bytes > 512, so under the old captail_500was unreachable in that case. Order of attrs in the new test matches the manager call exactly.src/pkg/fleetreport/runtime.go:128,640— the tracker re-bounds at 8 KiB keeping the newest bytes, so 4096 stays inside the storage bound and the tail survives there too.- Changelog fragment name
fixed-15311-…matches thechangelog.dguard convention; commit is DCO-signed.
One pre-existing thing, not a blocker and not introduced here: the final out[:maxDiagnosticsBytes] at L205-206 slices by byte and can cut a multi-byte rune in half. With the larger cap it will fire far less often, but if someone wants it clean later, strings.ToValidUTF8 or a rune-aware cut would do.
Fork PR, so CI is still pending approval; nothing in the change itself needs a maintainer decision.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Ready for a human merge: verdict approved @ebe3d72, 54/54 checks green, DCO yes, soak 30m0s elapsed. Blocked only by The hive never approves on your behalf: |
|
Test coverage (quality) — approve; one advisory, read at head ebe3d72.
func TestDiagnosticsStaysUnderCapOnOversizedPaneExcerpts(t *testing.T) {
logger, seen, _ := newFeed(t)
big := strings.Repeat("𝄞", 500) // 4 bytes/rune → 2000 bytes each; head+tail+flags > 4096
logger.Warn("pane content at timeout", "agent", "quality", "head_500", big, "tail_500", big)
d := (*seen)[0].diagnostics
if len(d) > maxDiagnosticsBytes {
t.Fatalf("diagnostics = %d bytes, cap %d", len(d), maxDiagnosticsBytes)
}
}Related, worth a glance while here: 🐝 Hive Agent: — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
stage 080-awaiting-merge-admission: no activity for 30m0s: merge-train:queued-behind#15315:2/3:0/0:fleetreport.go,slogfeed_test.go |
There was a problem hiding this comment.
Approved by @clubanderson for Hive auto-merge on green CI.
Approved from the Hive dashboard by @clubanderson as the owner decision on a Ready-for-human-merge pull request; the hive's merge sweep merges it on green CI.
|
@hivecommons-hive[bot]: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Hive queued this PR but could not set the |
|
Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict approve, @clubanderson approved this head as the owner decision ( |
Summary
The
slogfeedWARN/ERROR tee that feeds fleet self-reports (src/pkg/fleetreport/slogfeed/slogfeed.go) bounds the attribute excerpt it attaches to an observation at 512 bytes, writing attrs in record order and stopping at the first one that would cross the cap.agent.manager's"pane content at timeout"WARN (src/crux/agent/manager.go) logshead_500beforetail_500, each up to 500 runes — and worse when the pane is full of multi-byte box-drawing characters (❯,╰─, …).head_500alone routinely fills the 512-byte budget, sotail_500— the pane's current state, and the only field that actually helps diagnose a stuck CLI — was silently dropped every single time this warning fired.Both #14586 and #15311 self-reported this exact WARN with no usable excerpt attached; #14586 was closed uninvestigated specifically for lack of one ("No pane excerpt (head_500/tail_500) was attached ... if it recurs, attach the 'pane content at timeout' log").
Fix
Raise
maxDiagnosticsBytesfrom 512 to 4096 — comfortably fits both 500-rune excerpts plus the preceding flags with room to spare, while staying well under the tracker's own 8KiB storage bound (fleetreport.maxDiagnosticsBytes).Added a regression test (
TestDiagnosticsKeepsBothPaneTimeoutExcerpts) that reproduces the shape of the real WARN (worst-case multi-bytehead_500+ a markedtail_500) and assertstail_500survives.Test plan
go test ./pkg/fleetreport/slogfeed/...) — localgo test/go buildare blocked for agents perAGENTS.md; relying on CI per that doc.gofmt -lclean on changed files.head_500alone at worst case ≈1500 bytes already exceeds 512).Closes #15311
— hive: backend=claude model=claude-sonnet-5-5 effort=medium