Skip to content

🐛 Meetings: don't restart-loop the Spektacular stream child when live mode isn't available - #18633

Closed
clubanderson wants to merge 1 commit into
v5from
fix/18630-spek-stream-restart-loop
Closed

clubanderson wants to merge 1 commit into
v5from
fix/18630-spek-stream-restart-loop

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #18630

Root cause

The hive image pins spektacular 0.31.0, which has no meeting command (live meeting mode is hivecommons/spektacular#126, still open). Each live meeting's spektacular meeting --from stdin ... child printed cobra's unknown command "meeting" inside the CLI's indented JSON error envelope on stdout and exited 1. spektacular.Stream restarted it with backoff (1s, 2s, 4s, 8s, 16s), logging [spektacular] stream child exited; restarting at WARN each time: 5 in 10 minutes, which the slog feed filed as unclassified_error_repeated. Pod logs on the reporting hive show exactly that series, plus six spek wrote a non-JSON line warnings per restart (the envelope lines) and forwarding to spek ... stream child is not running for transcript lines.

Fix

  • spektacular.StreamOptions.Probe: the meetings wiring probes spektacular meeting --help once per installed binary (cached by path, size and mtime). An unknown-command answer marks the stream Unsupported: Start returns nil with the stream already done, no child is started or restarted, and one INFO line is logged per binary. As a fallback, a child whose own exit output says unknown command is not restarted either.
  • A supported child that keeps exiting still restarts with capped exponential backoff and the restart cap; the restart line is WARN at most once per BackoffMax window (INFO otherwise) and is classified by the new log-fed class spek_stream_unavailable instead of unclassified_error_repeated.
  • The live session treats an unsupported Spek as off from the start (the Meetings panel keeps saying notes and clarifying questions are off until spektacular#126), stops forwarding lines and answers once Spek is gone, and warns about a failed forward only on the transition to paused rather than per line.

Tests

  • pkg/spektacular: probe-unsupported never starts a child and probes once; unknown-command exit is not restarted; probe-supported starts normally; restart WARN once per backoff window.
  • pkg/meetings/live: unsupported Spek gets no forwarded lines and no warnings; forwarding failure warns once per outage.
  • pkg/fleetreport: the new class in the threshold and log-fed classification tables.

Changelog: changelog.d/fixed-18630-spek-stream-restart-loop.md. Docs: class row in src/docs/fleet-report.md.

🤖 Generated with Claude Code

… mode isn't available

The pinned spektacular (0.31.0) has no `meeting` command
(hivecommons/spektacular#126 is unreleased), so every live meeting child
printed cobra's unknown-command envelope and exited 1. The stream
restarted it five times per meeting, each restart a WARN, which the
slog feed self-reported as unclassified_error_repeated (#18630), and the
session warned once per transcript line it could not forward.

- spektacular.Stream gains an optional Probe (`meeting --help`), run
  once per installed binary; an unknown command marks the stream
  Unsupported: no child is started or restarted, one INFO line per
  binary. A child whose own exit output says unknown command is not
  restarted either.
- Restart lines are WARN at most once per BackoffMax window (INFO
  otherwise) and map to a new log-fed class spek_stream_unavailable.
- The live session treats an unsupported Spek as off from the start
  (panel status unchanged), stops forwarding once Spek is gone, and warns
  about a failed forward only on the transition to paused.

Fixes #18630

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18625 ✨ Betty names who and what each answer is for when requests overlap same files src/pkg/meetings/live/session.go
#18623 ✨ Betty runs several model calls per meeting at once same files src/cmd/hive/meetingswire.go
#18627 ✨ Betty speaks ready answers shortest first and moves a long voice queue to chat CONFLICT src/cmd/hive/meetingswire.go
src/pkg/meetings/live/session.go (conflict)
#18629 ✨ Betty times every answer stage and answers live-state questions without waiting for search same files src/pkg/meetings/live/session.go

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/pkg/spektacular/stream.go 36 ARITHMETIC_BASE

Verdict: uncovered (#17349).

@clubanderson

Copy link
Copy Markdown
Member Author

Combined into #18635 to get through CI and review together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[selfreport] unclassified_error_repeated: [spektacular] stream child exited; restarting

1 participant