Skip to content

✨ feat(hub): carry bounded activity events through the heartbeat and hub ring - #16049

Merged
hivecommons-hive[bot] merged 6 commits into
v5from
scanner/fix-15964
Oct 31, 2026
Merged

hivecommons-hive[bot] merged 6 commits into
v5from
scanner/fix-15964

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #15964
Part of #15861 (RFC by @hanthor; follows #15951 and #15963)

Why

The hub Activity Monitor from #15951/#15963 shows where each item is now, but nothing about how it got there. RFC #15861 §A/§B asks the hub board to animate stage hops and show merge toasts like the spoke Activity Monitor does. It also requires that this stays bounded, disclosed, memory-only and polled by ETag (no SSE), and that a viewer never learns about items they are not authorized to see. Without events in the activity block, the hub can only diff whole digests. Diffing whole digests loses merges, because a merged PR just disappears from the item list, and it misses any hop that happens between two polls.

What shipped (in this PR)

  • Contract (pkg/hubactivity): the activity block has new events (opened/stage/merged, each with a monotonic seq) and events_dropped fields. Events are excluded from the content etag, and an unchanged marker can carry them, so the existing etag handshake is unchanged.
  • Spoke: the publisher compares each tracker snapshot with the previous one to find hops and opens, and reads merges from the tracker's recently_merged list. The first snapshot only sets the baseline. It buffers up to 200 events, drops the oldest first and reports the count in events_dropped. Events are trimmed only when a response's new activity_events_ack covers them. Private-repo items produce no events unless share_private is on. heartbeat_omit also applies to events.
  • Hub: a fleet-wide in-memory ring holds the last 1,000 events per hive. It removes duplicates by acknowledged seq and enforces the 200-per-block cap again. It is cleared when a hive is forgotten. GET /api/hub/activity returns events[] only for hives the viewer is authorized for, and copies them out per hive before aggregation. Events are part of the response body, so new events change the ETag.
  • Tests for both sides: the 200 and 1,000 caps, ack-based trimming, per-user scoping and private exclusion, and a merge appearing in /api/hub/activity after one heartbeat.
  • Docs (hub-server.md, telemetry.md) and a changelog fragment.

Status

The PR is open and CI has not finished. Nothing is deployed or activated. Hubs and spokes only exchange events once both run this code. Older hubs ignore events, and older spokes never send them. Against an older hub, a spoke's buffer stays capped at 200 because it never receives an ack.

Overlap with #16044

#16044 moves the s.activity.record/forget call sites in server.go and adds a test at the end of activity_test.go. This PR leaves those lines alone. The changes are limited to the store internals in activity.go, one added resp.ActivityEventsAck line near resp.ActivityETag, and new files. Event purging is inside activityStore.forget, so it follows wherever #16044 moves the call.

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@hanthor

…hub ring

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 31, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 31, 2026
@hivecommons-hive

hivecommons-hive Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: pushed f0a524d — NormalizeEvent now lower-cases Type/Kind before the allow-list check, which is what TestNormalizeEvent_BoundsAndValidates asserts. CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review (read at head 53d5788, which supersedes the dispatched 9da07a3). No in-scope blocker; one advisory edge case.

Low — spoke clock step-back across a restart silently loses events. src/pkg/hub/spoke/activity_publisher.go:149 seeds nextSeq from time.Now().UnixMilli()*1000 so a restarted spoke numbers past the hub's ack. If the spoke's wall clock has stepped backwards before the restart (VM resume, NTP correction), every new Seq is ≤ the hub's lastSeq, so activityEventRing.ingest (src/pkg/hub/activity_events.go:62-64) drops them all and eventsAck still returns the old high mark, which acknowledgeEventsLocked (activity_publisher.go:231) treats as delivered and trims. Events vanish without any events_dropped disclosure until the clock passes the previous maximum. Bounded by the skew and not a contract blocker; a cheap hardening would be to have the hub reset lastSeq when a batch's lowest Seq is far below it, or disclose the discarded count in dropped.

Verified: ring is only touched under activityStore.mu (activity.go:80-92); Sanitize omits Events so the content etag/handshake is unchanged (hubactivity.go:164); spoke holds pub.mu from build through ack (spoke/heartbeat.go:1797-1801, :1877-1878); tests cover the 200/1,000 caps, ack trimming, per-user scoping, private exclusion and merge-in-one-heartbeat.

Review findings from reviewer (folded here because this head already carries its hive review):

CI is red for a real reason in this PR's own test, and it is deterministic — it failed identically on all three reruns of test (rest 2/4).

src/pkg/hubactivity/events_test.go:16-17 feeds From: "Needs Review!" and Stage: "Actionable" into NormalizeEvent, then at :33-34 asserts got.From != "" and got.Stage != "". But slug() (src/pkg/hubactivity/hubactivity.go:298-304) is a validator, not a normalizer: it trims and returns the input only if it already matches slugPattern = ^[a-z0-9][a-z0-9_.-]{0,63}$ (hubactivity.go:52), else "". Neither test input is lowercase, so both become "" and the test fails:

events_test.go:34: from/stage not slugged: "" ""
--- FAIL: TestNormalizeEvent_BoundsAndValidates
FAIL	github.com/hivecommons/hive/pkg/hubactivity

This is a test bug, not a production bug: real From/Stage values come from Item.Stage via observeLocked (src/pkg/hub/spoke/activity_publisher.go:270-280), which is already slug-shaped. Fix is to use slug-shaped inputs (e.g. From: "needs-review", Stage: "actionable") or to assert the drop behaviour for non-slugs explicitly. The coverage and aggregate test jobs fail only as a consequence of this shard.

Everything else I read checks out against the tree:

  • Sanitize (hubactivity.go:164) builds a fresh Digest without Events, and ContentETag (hubactivity.go:197-202) hashes only totals/items/truncated, so events riding on an unchanged marker neither pollute the stored digest nor break the etag handshake — the comment at hubactivity.go:100-105 is accurate.
  • Spoke restart is handled: nextSeq is seeded from wall-clock ms ×1000 (activity_publisher.go:149), so a restarted spoke numbers past the hub's lastSeq instead of being silently deduped by ingest's e.Seq <= h.lastSeq check (src/pkg/hub/activity_events.go:66). Hub restart is handled the other way (ack 0 → spoke keeps and resends, activity_publisher.go:228-231).
  • eventsSnapshot is restricted to the already-authorized hive ids and addActivityEvents re-checks visible (activity_events.go:131-145, 183-188), so per-hive authorization holds for events as it does for digests.
  • redactActivity strips events under heartbeat_omit: repos and blanks titles under task_titles (activity_publisher.go:400-417).
  • forget/forgetOld on the ring are wired into activityStore.forget and forgetOldLocked (activity.go:132, 141) so a removed hive's events go with its digest.

Once the test input is fixed this should be mergeable; I have no other findings.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the tree at f0a524d (hub store, spoke publisher, dashboard snapshot, hubactivity contract) and the failing test (rest 2/4) shard. The production code looks correct to me; the red shard is a test-input problem.

Why CI is red — src/pkg/hubactivity/events_test.go:34

TestNormalizeEvent_BoundsAndValidates feeds From: "Needs Review!" and Stage: "Actionable" and then asserts got.From != "" / got.Stage != "". But slug (hubactivity.go:298-304) is a validator, not a slugifier: it returns the input only when it matches slugPattern = ^[a-z0-9][a-z0-9_.-]{0,63}$ (hubactivity.go:52) and "" otherwise. Neither input matches (capitals, space, !), so NormalizeEvent correctly empties both and the test fails with from/stage not slugged: "" "". Fix is in the test: use slug-shaped inputs ("needs-review", "actionable") — production callers already pass slugs (dashboard/hub_activity.go:110 maps it.StageSlug). Severity low; coverage/test are red only because this shard failed.

What I verified as correct

  • Events never reach a stored digest twice: hubactivity.Sanitize (hubactivity.go:164-165) does not copy Events, and ContentETag (hubactivity.go:197-202) hashes only totals/items/truncated, so the existing etag handshake is unchanged as the body claims.
  • Resend idempotency: activityEventRing.ingest (hub/activity_events.go:51-79) skips Seq <= lastSeq and only adds dropped when fresh > 0; the spoke holds p.mu across build→post→ack (spoke/activity_publisher.go:356-383), so the normal path acks each build before the next one.
  • Authorization: eventsSnapshot copies only the authorized ids and addActivityEvents re-checks visible[e.hiveID] (hub/activity_events.go:124-141, 176-200); forget/forgetOld clear the ring alongside the digest (hub/activity.go:132, 141).
  • Private gating on the spoke: observeLocked skips it.Private && !sharePrivate for both opens/hops and merges (spoke/activity_publisher.go:264-266, 282); redactActivity handles repos/task_titles for events (spoke/activity_publisher.go:405-417).
  • Payload bound: 200 events × ≤200-rune titles stays well under the hub's 1 MB maxPayloadBytes (hub/server.go:126), including the resend-forever case against a pre-[scanner] Carry bounded activity events through the heartbeat and hub ring for fleet stage hops and merge toasts #15964 hub.

Body/code mismatch (wording only)

The PR body says the hub ring "holds the last 1,000 events per hive"; the code and changelog say fleet-wide (activityEventRingSize = 1000, hub/activity_events.go:22, with the oldest dropped across all hives). Worth correcting the body so the operator-facing expectation matches.

Related: #16044 moves the record/forget call sites; this PR deliberately leaves those lines alone and the purge lives inside activityStore.forget, so the two should rebase cleanly.

Evidence bundle `#16049a597761` sha256:8c686a653fd0c9bb8d7408e8590e330747908c41b02958e6dc9d5ea9a2abcce4 — download from the dashboard or `hivectl review evidence #16049`.

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 31, 2026
The coverage gate failed with hubactivity at 76.3% (floor 90%) because
events.go landed without tests. Cover NormalizeEvent (bounding, UTC
normalization, every rejection branch), SanitizeEvents (dedupe by Seq,
ordering, MaxEvents cap with oldest-dropped disclosure, input not
modified) and ClampCount.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 31, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective (reviewed at head 53d57884; the dispatched head 9da07a3e had already been superseded).

No in-scope blocker. Event ingest sits behind the same per-hive bearer as the digest (pkg/hub/activity_update.go:62-73, pkg/hub/server.go:2737), hubactivity.Sanitize builds a fresh digest so raw spoke events never reach the stored block (pkg/hubactivity/hubactivity.go:164-189), and eventsSnapshot(ids) is only called with the ids visibleActivityHives already authorized (pkg/hub/activity.go:371-377, pkg/hub/activity_events.go:120-137). Two low notes:

  1. Private events outlive a share_private opt-out (low, in-scope). observeLocked skips private items only at creation time (pkg/hub/spoke/activity_publisher.go:265, :283); events already buffered while share_private was on stay in p.events and are resent on every build (:177) until an ack covers them (:228). Against a pre-#15964 hub that never acks, they are re-sent indefinitely after the operator turned sharing off. Suggested fix: in build, when !opts.SharePrivate, drop buffered events with Private before cloning them into d.Events. TestActivityEventsExcludePrivateItemsUnlessShared (activity_events_test.go:166) only covers a fixed setting, not the toggle.

  2. One spoke can evict the whole fleet's events (low, out-of-scope — the fleet-wide 1,000 ring is what #15964 specifies). The ring is shared across hives and trimmed oldest-first (pkg/hub/activity_events.go:23, :79-80); a single authenticated spoke sending 200 fresh-seq events per 15 s update empties everyone else's events in ~5 updates. Items have no equivalent shared resource (per-hive 250 ceiling). A per-hive fair-share cap inside the ring would close this; backlog material, not a blocker here.

Minor: the PR body says the hub ring holds "the last 1,000 events per hive"; the code and the changelog fragment say fleet-wide (activity_events.go:21-23). The code matches the issue; the body wording is off.

— hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

test (rest 2/4) is red because of this PR's own diff (FIX_DIFF), not the base.

pkg/hubactivity.TestNormalizeEvent_BoundsAndValidates fails deterministically — 3/3 runs incl. two re-runs (job 124559286082 @ 53d5788):

events_test.go:25: expected event to be accepted

Root cause: the test feeds Type: " STAGE\t" / Kind: "PR" and asserts they come back as stage / pr, but NormalizeEvent runs them through cleanToken (trim + control-char strip only — no case folding) before oneOf(..., allowedEventTypes). "STAGE" is not in {opened, stage, merged}, so out.Type == "" and the event is rejected at events.go:70. Reproduced locally at 53d5788: NormalizeEvent{Type:" STAGE\t", Kind:"PR"} → ok=false; identical event with "stage"/"pr" → ok=true.

One-line fix matching the test's intent (tokens are case-insensitive on the wire):

// pkg/hubactivity/events.go, NormalizeEvent
out.Type = oneOf(strings.ToLower(cleanToken(e.Type)), allowedEventTypes)
...
out.Kind = oneOf(strings.ToLower(cleanToken(e.Kind)), allowedKinds)

If uppercase tokens are instead meant to be rejected, change the test fixture to "stage"/"pr" and add an explicit "upper-case type" case to TestNormalizeEvent_RejectsMalformed so the policy is asserted either way.

The other red checks on this head (Sign-off survives the squash, native-queue-policy) are cancelled superseded runs, not failures.

— quality agent (did not push: scanner/ branch)


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

…e validation

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 31, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness delta re-review (53d5788 → f0a524d; read src/pkg/hubactivity/events.go, src/pkg/hub/activity_events.go, src/pkg/hub/spoke/activity_publisher.go, src/pkg/hub/activity_events_test.go at head).

  • Fixed: NormalizeEvent now case-folds Type/Kind (events.go:69,74), so TestNormalizeEvent_BoundsAndValidates (events_test.go:13-27, " STAGE\t"/"PR" → stage/pr) passes.
  • Still present (in-scope, low): hub events_dropped double-counts a spoke's cumulative disclosure. The spoke's eventsDropped is cumulative until an ack covers sentMaxSeq (activity_publisher.go:178-183, 232-236), so a resend after a lost ack carries the same drops again. The hub adds it to its running total whenever the batch has any fresh event (activity_events.go:66-78), contradicting its own comment at :49-51 ("…or its drops counted twice"). TestActivityEventRing_DedupAndAck now bakes this in: the spoke drops 3 events, resends with EventsDropped = 3 plus one new event, and the test asserts 6 (activity_events_test.go:137-156). Fix: track the spoke's last disclosed count per hive and add only the increase over it (or reset the hub's per-hive count to the disclosed value), and flip the test expectation to 3.
  • docs-currency note (old-hub re-send behaviour, activity_publisher.go:365) is outside this perspective and the delta does not touch that file.

— hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

The spoke's events_dropped is cumulative until an ack covers its latest
build, so a batch resent after a lost ack restates the drops the hub
already counted. The ring now records the count each counted batch
claimed per hive and, when the batch holds an already-acknowledged event
(proof the spoke never saw the ack), adds only the increase over it; a
batch of only new events marks a spoke that trimmed and reset, so its
disclosure counts in full.

The dedup test now asserts 3 drops for 3 real drops across the resend,
and covers a grown resend and a post-ack batch.

Hive-Fix-Attempt: 1/6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 31, 2026
@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 31, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness delta re-review (read f0a524d…e3dbf24: src/pkg/hub/activity_events.go, activity_events_test.go, plus the spoke publisher and hubactivity at head).

Still present — src/pkg/hubactivity/events_test.go:18-19,33 makes test (rest 2/4) red at this head. The delta did not touch this file. TestNormalizeEvent_BoundsAndValidates feeds From: "Needs Review!" / Stage: "Actionable" and asserts got.From != "" / got.Stage != "", but NormalizeEvent routes both through slug() (events.go:76-77), which is a validator: it returns the input only when it matches slugPattern = ^[a-z0-9][a-z0-9_.-]{0,63}$ (hubactivity.go:52, :298-304) and "" otherwise. Reproduced against the head package: ok=true type="stage" kind="pr" from="" stage="". The f0a524d fix (lower-casing Type/Kind) addressed the :25 failure, not this one. Fix in the test: use slug-shaped inputs ("needs-review", "actionable"), or assert the empty-on-non-slug behaviour explicitly. Production callers already pass slugs.

Delta — restated drop accounting (activity_events.go:72-94) is correct for the case it tests; one bounded gap. restated is inferred from a duplicate Seq (:74-79), which matches the spoke's semantics: eventsDropped is cumulative and only reset when an ack covers sentMaxSeq (spoke/activity_publisher.go:228-236). If an ack is lost and the spoke then buffers more than MaxEvents (200) new events before its next accepted beat, the resend contains no previously-seen Seq (the old ones were trimmed at :309-311), so restated stays false and the still-cumulative disclosure is added in full — over-counting the earlier batch's drops once. Disclosure-only, bounded by one batch's count, and undetectable from the hub side without more wire state, so advisory; worth a comment at :74 and a test if you want the limit on record.

Previous intent-alignment note (PR body "1,000 events per hive" vs fleet-wide ring at activity_events.go:21-23) is unchanged; not re-filed here.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

slug() only accepts lowercase [a-z0-9][a-z0-9_.-]* so "Needs Review!" and
"Actionable" correctly normalize to ""; the test asserted the opposite.
Exercise trimming with padded valid slugs and assert that invalid slugs
blank the field without dropping the event.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 31, 2026
…turnover

A resend whose earlier events were already trimmed for room carries no
repeated Seq, so the hub cannot tell the spoke's cumulative disclosure
restates drops already counted and adds them once more. Document the
limit at the detection site and pin the bounded over-count in a test.

Hive-Fix-Attempt: 2/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 31, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 82b98a9 into v5 Oct 31, 2026
92 of 101 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-15964 branch October 31, 2026 15:15
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. stage/050-review-pending Review swarm requested; awaiting a verdict for the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] Carry bounded activity events through the heartbeat and hub ring for fleet stage hops and merge toasts

0 participants