Skip to content

fix: stop dropping tail_500 from pane-timeout self-report diagnostics - #15319

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v5from
onkar717:fix/agent-pane-timeout-content
Oct 29, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v5from
onkar717:fix/agent-pane-timeout-content

Conversation

@onkar717

Copy link
Copy Markdown
Contributor

Summary

The slogfeed WARN/ERROR tee that feeds fleet self-reports (src/pkg/fleetreport/slogfeed/slogfeed.go) bounds the attribute excerpt it attaches to an observation at 512 bytes, writing attrs in record order and stopping at the first one that would cross the cap.

agent.manager's "pane content at timeout" WARN (src/crux/agent/manager.go) logs head_500 before tail_500, each up to 500 runes — and worse when the pane is full of multi-byte box-drawing characters (❯, ╰─, …). head_500 alone routinely fills the 512-byte budget, so tail_500 — the pane's current state, and the only field that actually helps diagnose a stuck CLI — was silently dropped every single time this warning fired.

Both #14586 and #15311 self-reported this exact WARN with no usable excerpt attached; #14586 was closed uninvestigated specifically for lack of one ("No pane excerpt (head_500/tail_500) was attached ... if it recurs, attach the 'pane content at timeout' log").

Fix

Raise maxDiagnosticsBytes from 512 to 4096 — comfortably fits both 500-rune excerpts plus the preceding flags with room to spare, while staying well under the tracker's own 8KiB storage bound (fleetreport.maxDiagnosticsBytes).

Added a regression test (TestDiagnosticsKeepsBothPaneTimeoutExcerpts) that reproduces the shape of the real WARN (worst-case multi-byte head_500 + a marked tail_500) and asserts tail_500 survives.

Test plan

  • CI (go test ./pkg/fleetreport/slogfeed/...) — local go test/go build are blocked for agents per AGENTS.md; relying on CI per that doc.
  • gofmt -l clean on changed files.
  • New test fails against the old 512-byte cap and passes at 4096 (verified by inspection of the truncation logic: head_500 alone at worst case ≈1500 bytes already exceeds 512).

Closes #15311

— hive: backend=claude model=claude-sonnet-5-5 effort=medium

The slogfeed WARN/ERROR tee that feeds fleet self-reports bounds the
attribute excerpt it attaches to an observation at 512 bytes, writing
attrs in record order and stopping at the first one that would cross
the cap. agent.manager's "pane content at timeout" WARN logs head_500
before tail_500, each up to 500 runes (worse with multi-byte
box-drawing characters like ❯ and ╰─); head_500 alone routinely fills
512 bytes, so tail_500 — the pane's CURRENT state, and the only field
useful for diagnosing a stuck CLI — was dropped every single time.

Both hivecommons#14586 and hivecommons#15311 self-reported this exact WARN
with no usable excerpt attached, and hivecommons#14586 was closed uninvestigated
for lack of one. Raise the cap to 4096 bytes, comfortably fitting both
excerpts plus the preceding flags, well under the tracker's own 8KiB
storage bound.

Closes hivecommons#15311

Hive-Run: hivecommons#15311
Hive-Plan: fix pane-timeout selfreport diagnostics truncation
Hive-Spec: selfreport-diagnostics#agent_pane_timeout
Signed-off-by: onkar717 <shelkeonka@gmail.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@onkar717 this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • build-and-test
  • golangci-lint

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15374 🐛 fix: suppress gateway errors in log messages same files src/pkg/fleetreport/slogfeed/slogfeed_test.go

…nt-pane-timeout-content

Signed-off-by: onkar717 <shelkeonka@gmail.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. The mechanism described is real and the fix is the minimal one.

What I verified at ebe3d72, reading the tree rather than the diff:

  • src/pkg/fleetreport/slogfeed/slogfeed.go:181-208 — diagnostics() writes attrs in record order and write() refuses once b.Len() >= maxDiagnosticsBytes (L186), then hard-truncates at L205. So an attr that fills the budget drops every attr after it, not just the overflow.
  • src/crux/agent/manager.go:1275-1284 — the WARN logs head_500 before tail_500, each truncate*(output, 500) runes. 500 × 3-byte box-drawing runes = 1500 bytes > 512, so under the old cap tail_500 was unreachable in that case. Order of attrs in the new test matches the manager call exactly.
  • src/pkg/fleetreport/runtime.go:128,640 — the tracker re-bounds at 8 KiB keeping the newest bytes, so 4096 stays inside the storage bound and the tail survives there too.
  • Changelog fragment name fixed-15311-… matches the changelog.d guard convention; commit is DCO-signed.

One pre-existing thing, not a blocker and not introduced here: the final out[:maxDiagnosticsBytes] at L205-206 slices by byte and can cut a multi-byte rune in half. With the larger cap it will fire far less often, but if someone wants it clean later, strings.ToValidUTF8 or a rune-aware cut would do.

Fork PR, so CI is still pending approval; nothing in the change itself needs a maintainer decision.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Ready for a human merge: verdict approved @ebe3d72, 54/54 checks green, DCO yes, soak 30m0s elapsed. Blocked only by role/newcomer (below require_tiers).

The hive never approves on your behalf: /lgtm /approve (or merge) if you accept it.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test coverage (quality) — approve; one advisory, read at head ebe3d72.

TestDiagnosticsKeepsBothPaneTimeoutExcerpts fails at the old cap (head_500 alone is ~1500 bytes, so tail_500 was never written) and passes at 4096 — it pins the change. What nothing pins is the other side of the cap: no test in slogfeed_test.go references maxDiagnosticsBytes, so the b.Len() >= maxDiagnosticsBytes stop in write and the final out = out[:maxDiagnosticsBytes] (slogfeed.go:199,218-219) are exercised only incidentally. Since this PR is the one raising the cap and documenting what must fit under it, a companion case that overflows it would make the bound a tested contract rather than a comment:

func TestDiagnosticsStaysUnderCapOnOversizedPaneExcerpts(t *testing.T) {
	logger, seen, _ := newFeed(t)
	big := strings.Repeat("𝄞", 500) // 4 bytes/rune → 2000 bytes each; head+tail+flags > 4096
	logger.Warn("pane content at timeout", "agent", "quality", "head_500", big, "tail_500", big)
	d := (*seen)[0].diagnostics
	if len(d) > maxDiagnosticsBytes {
		t.Fatalf("diagnostics = %d bytes, cap %d", len(d), maxDiagnosticsBytes)
	}
}

Related, worth a glance while here: out[:maxDiagnosticsBytes] slices bytes, so when the excerpt is cut inside a multi-byte rune — exactly the box-drawing input this PR is about — the stored diagnostics end in an invalid UTF-8 tail. Pre-existing at 512, not introduced here; if you add the case above, utf8.ValidString(d) would tell you whether the tracker's second bound already repairs it. Not blocking either way.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission: no activity for 30m0s: merge-train:queued-behind#15315:2/3:0/0:fleetreport.go,slogfeed_test.go

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @clubanderson for Hive auto-merge on green CI.

Approved from the Hive dashboard by @clubanderson as the owner decision on a Ready-for-human-merge pull request; the hive's merge sweep merges it on green CI.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@hivecommons-hive[bot]: changing LGTM is restricted to collaborators

Details

In response to this:

Approved by @clubanderson for Hive auto-merge on green CI.

Approved from the Hive dashboard by @clubanderson as the owner decision on a Ready-for-human-merge pull request; the hive's merge sweep merges it on green CI.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hivecommons-hive hivecommons-hive Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive queued this PR but could not set the prow/lgtm commit status, so Prow may strip the lgtm label and the auto-merge queue may not hold.

@hivecommons-hive
hivecommons-hive Bot merged commit c6755ac into hivecommons:v5 Oct 29, 2026
63 of 64 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#15319ebe3d72` sha256:9370d0e9187dc5e9befddc586fc4a5aa606b1def3b1fac5a23c989356a24c51d — download from the dashboard or `hivectl review evidence #15319`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict approve, @clubanderson approved this head as the owner decision (admitted_by: operator-approval:clubanderson) — merged as-is on that approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[selfreport] agent_pane_timeout: pane content at timeout

1 participant