Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-activity-event-retention-bound.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The hub activity event ring now also drops any event held longer than 24 hours, so a low-volume hive that keeps heartbeating no longer keeps old events in hub memory indefinitely after the fair-share eviction change ([#16072](https://github.com/hivecommons/hive/issues/16072)).
2 changes: 1 addition & 1 deletion src/docs/api-reference/hub-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f
| `GET` | `/api/fleet-stats` | Hub handler-specific | Fleet Stats | `pkg/hub/server.go:1723` |
| `GET` | `/api/hub/evidence` | Hub admin | Read-only cross-hive evidence drill-down from each spoke's stored heartbeat `evidence` summary ([#12116](https://github.com/hivecommons/hive/issues/12116)); non-admins get `403`. Returns `spokes`, `reporting`, pooled `calibration` (bucket 0–5 → `{n, bad, rate, lo, hi}`; the hub view folds the six buckets into four reviewer-confidence ranges), `findings[]` (the plain-language *What this says* sentences, generated from the same fields; [#12456](https://github.com/hivecommons/hive/issues/12456)), `by_acmm_level` (`level`, `hives`, `merged_90d`, `revert_rate`, and `time_to_merge` — `median_of_hive_medians_s`, `hives`, `n` — the median of the level's per-hive median time-to-merge, omitted when no hive at the level reports one; [#12464](https://github.com/hivecommons/hive/issues/12464)), `family_false_approve_rate` (`same`/`cross`/`mixed`, hives classed by their reviewer vs author family sets), fleet `record_coverage` (merged PRs with an evidence record over `prs_merged_90d`), fleet panels ([#12370](https://github.com/hivecommons/hive/issues/12370)) `outcomes_by_author_kind[]` (`kind`, `hives`, `n`, `merged`, `reverted`, `merge_rate`, `revert_rate`), `write_counters[]` (`tier`, `hives`, `attempted`, `blocked`, `block_rate`) and `shadow_gate` (`review_gate` / `intent` → `held` / `passed` outcomes with `revert_rate`), each omitted while no spoke reports it and repeated per `by_acmm_level[]` row, and per hive `hives[]` with `acmm_level`, `prs_merged_90d`, `record_coverage`, `family_mix`, false-approve / false-block / revert rates, families, a calibration series the raw `counts` behind them (including `by_author_kind`) so the drill-down can re-pool a hive / ACMM-level selection client-side, and `criteria[]`: the per-axis criteria table derived hub-side from the quadrant criteria (`axis`, `label`, `scored`, `criteria[]` with `name`, `label`, `status` `met`/`unmet`/`not_reported`, `display`/`score` when measured, `reason` otherwise; offline, evidence-less or unscored hives are `not_reported`, never `unmet`; [#12463](https://github.com/hivecommons/hive/issues/12463)). Every rate is `{n, rate, lo, hi}` with a 95% Wilson interval; `rate`/`lo`/`hi` are null when `n` < 10. Window selection ([RFC #12062](https://github.com/hivecommons/hive/issues/12062)): `?hours=24|168|720` pools each hive's matching heartbeat `windows` block instead of its 90-day block (any other or absent `hours` → 90 days); `?from=YYYY-MM-DD|RFC3339[&to=…]` (date-only `to` is inclusive, `to` defaults to now, `to` ≥ `from`, span ≤ 366 days, else `400`) is served from the smallest reported window that reaches back to `from` and flagged `window.approximate`. The response carries `window` (`hours`, `key`, `label`, `from`, `to`, `approximate`, `coverage_available` — `record_coverage` is only computed for the 90-day window because `prs_merged_90d` is a 90-day count), `window_unreported` and per-hive `window_unreported` for hives whose spoke predates per-window reporting. `?hive=<id>[,<id>…]` (repeatable) and `?level=<n>` restrict the registry server-side to those hives / that ACMM level before pooling. Hub admins open it from the Fleet quadrant card or its **🔬 Evidence** button ([#12117](https://github.com/hivecommons/hive/issues/12117)), pick a window (24h / 7d / 30d / 90d / custom), filter it by hive and ACMM level and read each section's ⓘ explainer; the per-hive author-kind revert counts behind the *Revert rate by author kind* panel are `hives[].counts.by_author_kind` ([#12326](https://github.com/hivecommons/hive/issues/12326)). Never consulted by merge decisions | `pkg/hub/evidence.go` |
| `GET` | `/api/hub/evidence/report` | Hub admin | Plain-language report of the hub evidence view ([#12456](https://github.com/hivecommons/hive/issues/12456)), rendered server-side from the same `EvidenceView` as `/api/hub/evidence` so every figure reconciles with the modal; non-admins get `403`. Same `hours=` / `from=` / `to=` window and `hive=` / `level=` filter parameters. `format=md` (default, `text/markdown`) or `format=html` (standalone page, inline CSS, no scripts or external assets, hive names escaped); any other value → `400`. `download=1` sets `Content-Disposition: attachment; filename="hive-evidence-<hive|fleet>-<from>-<to>.md|html"`, otherwise inline. Sections: Summary (the generated findings), What we measured and why, The numbers, How to read these, What is not reported yet and why, Grade per section, footer link to `review-evidence.md`. `Cache-Control: no-store`. Read-only; never consulted by merge decisions | `pkg/hub/evidence_report.go` |
| `GET` | `/api/hub/activity` | Signed-in user | Read-only hub-wide activity view ([#15951](https://github.com/hivecommons/hive/issues/15951), RFC [#15861](https://github.com/hivecommons/hive/issues/15861)) merged from each spoke's heartbeat `activity` digest, held in hub memory only. Authorization is per user and per hive before aggregation: hub admins see every registered hive; everyone else only hives they own or hold an unexpired grant for; anonymous → `401`. Returns `hives[]` (`hive_id`, `name`, `status` `ok`/`stale`/`not_reporting`/`missing`, `reporting`, `stale` once the block is older than two heartbeats, `last_heartbeat`, `received_at`, `generated_at`, `totals` — `items`, `by_stage`, `by_modifier`, counting private and truncated items too — `truncated` and `events_dropped`), `items[]` (`hive_id`, `repo`, `n`, `kind`, `title`, `author_kind`, `stage`, `stage_n`, `since`, `mods`, `private`) ordered needs-human/needs-decision, stalled, furthest stage, oldest, a response-level `truncated` (cap 2000), and `events[]` ([#15964](https://github.com/hivecommons/hive/issues/15964): `hive_id`, `received_at`, `seq`, `type` `opened`/`stage`/`merged`, `repo`, `n`, `kind`, `title`, `from`, `stage`, `at`, `private`), oldest first, from a fleet-wide in-memory ring of the last 1,000 events (when full, the oldest event of the hive holding the most events is dropped, counted in that hive's `events_dropped`, so one hive cannot evict the others' events). Events are filtered per user and per hive exactly like items, before aggregation, so an event for a hive the viewer cannot see is never delivered; `mod`/`author_kind` filters keep only events whose item is still on the hive's board. A spoke block carries at most 200 events; drops are disclosed per hive as `events_dropped`. New events change the `ETag`. Each digest carries at most 100 items by default and never more than the 250-item hub ceiling; private-repository items appear only when the spoke opts in. Filters: `hive`, `repo`, `stage`, `mod`, `author_kind`, `q` (title substring or `#<n>`), each ≤ 200 characters (else `400`). `ETag` + `If-None-Match` → `304`; `Cache-Control: private, no-cache`, `Vary: Cookie, Authorization`. No actions | `pkg/hub/activity.go` |
| `GET` | `/api/hub/activity` | Signed-in user | Read-only hub-wide activity view ([#15951](https://github.com/hivecommons/hive/issues/15951), RFC [#15861](https://github.com/hivecommons/hive/issues/15861)) merged from each spoke's heartbeat `activity` digest, held in hub memory only. Authorization is per user and per hive before aggregation: hub admins see every registered hive; everyone else only hives they own or hold an unexpired grant for; anonymous → `401`. Returns `hives[]` (`hive_id`, `name`, `status` `ok`/`stale`/`not_reporting`/`missing`, `reporting`, `stale` once the block is older than two heartbeats, `last_heartbeat`, `received_at`, `generated_at`, `totals` — `items`, `by_stage`, `by_modifier`, counting private and truncated items too — `truncated` and `events_dropped`), `items[]` (`hive_id`, `repo`, `n`, `kind`, `title`, `author_kind`, `stage`, `stage_n`, `since`, `mods`, `private`) ordered needs-human/needs-decision, stalled, furthest stage, oldest, a response-level `truncated` (cap 2000), and `events[]` ([#15964](https://github.com/hivecommons/hive/issues/15964): `hive_id`, `received_at`, `seq`, `type` `opened`/`stage`/`merged`, `repo`, `n`, `kind`, `title`, `from`, `stage`, `at`, `private`), oldest first, from a fleet-wide in-memory ring of at most 1,000 events, none held longer than 24 hours (ageing out is not counted in `events_dropped`; when full, the oldest event of the hive holding the most events is dropped, counted in that hive's `events_dropped`, so one hive cannot evict the others' events). Events are filtered per user and per hive exactly like items, before aggregation, so an event for a hive the viewer cannot see is never delivered; `mod`/`author_kind` filters keep only events whose item is still on the hive's board. A spoke block carries at most 200 events; drops are disclosed per hive as `events_dropped`. New events change the `ETag`. Each digest carries at most 100 items by default and never more than the 250-item hub ceiling; private-repository items appear only when the spoke opts in. Filters: `hive`, `repo`, `stage`, `mod`, `author_kind`, `q` (title substring or `#<n>`), each ≤ 200 characters (else `400`). `ETag` + `If-None-Match` → `304`; `Cache-Control: private, no-cache`, `Vary: Cookie, Authorization`. No actions | `pkg/hub/activity.go` |
| `GET` | `/api/hub/version` | Hub handler-specific | Hub Version | `pkg/hub/server.go:1731` |
| `DELETE` | `/api/hub/registry/{id}` | Hub handler-specific | Registry Delete | `pkg/hub/server.go:1741` |
| `POST` | `/api/contribute/register` | Hub handler-specific | Contribute Proxy | `pkg/hub/server.go:1742` |
Expand Down
2 changes: 1 addition & 1 deletion src/docs/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ hub:
share_private: false # private-repository items are dropped (still counted in totals) unless true
```

When the cap applies, items needing a human or a decision are kept first, then stalled ones, then the furthest stage, and the number left out is reported as `truncated`. A repository counts as public only when the hub lists the hive as public (`hub.is_public`, its project repos) or it is on `HIVE_PUBLIC_REPOS`. Once the hub acknowledges a digest's etag, later beats send only `etag` and `generated_at` until the tracker changes. A tracker change also sends an activity-only update over the heartbeat endpoint and bearer, at most one every 15 seconds; changes arriving meanwhile are merged into it. The block also carries `events` ([#15964](https://github.com/hivecommons/hive/issues/15964)): the stage hops, opens and merges the spoke saw since the hub last acknowledged them, derived by diffing successive tracker snapshots (the first one only seeds the baseline), each with a sequence number. At most 200 are buffered; the oldest are dropped first and `events_dropped` discloses how many. Events stay buffered, and are resent, until a heartbeat or update response's `activity_events_ack` covers them. Private-repository items yield no events unless `share_private` is on. The hub keeps the last 1,000 events across the fleet in memory only; when the ring is full it drops the oldest event of the hive holding the most events, so one hive cannot evict another hive's events. `hub.heartbeat_omit` applies: `repos` withholds every item and event (totals remain) and `task_titles` blanks titles.
When the cap applies, items needing a human or a decision are kept first, then stalled ones, then the furthest stage, and the number left out is reported as `truncated`. A repository counts as public only when the hub lists the hive as public (`hub.is_public`, its project repos) or it is on `HIVE_PUBLIC_REPOS`. Once the hub acknowledges a digest's etag, later beats send only `etag` and `generated_at` until the tracker changes. A tracker change also sends an activity-only update over the heartbeat endpoint and bearer, at most one every 15 seconds; changes arriving meanwhile are merged into it. The block also carries `events` ([#15964](https://github.com/hivecommons/hive/issues/15964)): the stage hops, opens and merges the spoke saw since the hub last acknowledged them, derived by diffing successive tracker snapshots (the first one only seeds the baseline), each with a sequence number. At most 200 are buffered; the oldest are dropped first and `events_dropped` discloses how many. Events stay buffered, and are resent, until a heartbeat or update response's `activity_events_ack` covers them. Private-repository items yield no events unless `share_private` is on. The hub keeps at most 1,000 events across the fleet in memory only, and no event is kept longer than 24 hours; when the ring is full it drops the oldest event of the hive holding the most events, so one hive cannot evict another hive's events. `hub.heartbeat_omit` applies: `repos` withholds every item and event (totals remain) and `task_titles` blanks titles.

## Telemetry agent

Expand Down
4 changes: 3 additions & 1 deletion src/pkg/hub/activity.go
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,9 @@ type ActivityHiveView struct {
// Truncated is how many of this hive's items its digest left out.
Truncated int `json:"truncated,omitempty"`
// EventsDropped is how many of this hive's events the spoke's or the
// hub's per-block cap dropped before they reached the ring (#15964).
// hub's per-block cap dropped before they reached the ring, plus those
// the full ring evicted (#15964). Events that aged out of the 24-hour
// retention window are not counted.
EventsDropped int `json:"events_dropped,omitempty"`
}

Expand Down
19 changes: 15 additions & 4 deletions src/pkg/hub/activity_events.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ import (

// activityEventRingSize bounds the fleet-wide ring. When it is full, the
// oldest event of the hive holding the most events is dropped first, so a
// single hive cannot evict the other hives' events.
// single hive cannot evict the other hives' events. Independently of that,
// an event ages out once it has been held for activityForgetAfter; ageing
// out is retention, not loss, so it is not counted as a drop.
const activityEventRingSize = 1000

// activityEvent is one stored event tagged with its hive.
Expand All @@ -34,8 +36,9 @@ type activityEvent struct {
type activityEventHive struct {
// lastSeq is the highest event sequence number accepted (the ack).
lastSeq uint64
// dropped sums the drops the spoke disclosed and the hub's own cap
// applied to its batches.
// dropped sums the drops the spoke disclosed, the hub's own cap
// applied to its batches and the ring's fair-share evictions. Events
// that age out after activityForgetAfter are not counted.
dropped int
// disclosed is the drop count the latest counted batch claimed (the
// spoke's disclosure plus the hub's cap). The spoke's disclosure is
Expand Down Expand Up @@ -157,13 +160,21 @@ func (r *activityEventRing) forget(hiveID string) {
r.entries = slices.DeleteFunc(r.entries, func(e activityEvent) bool { return e.hiveID == hiveID })
}

// forgetOld forgets hives that sent nothing for maxAge.
// forgetOld forgets hives that sent nothing for maxAge, and ages out any
// event held longer than maxAge from the hives that remain, so fair-share
// eviction never keeps a low-volume hive's events in memory indefinitely.
// Ageing out is not counted in dropped: every retained event eventually
// expires, so counting it would turn the loss counter into a running total
// and hide real cap drops.
func (r *activityEventRing) forgetOld(now time.Time, maxAge time.Duration) {
for id, h := range r.byHive {
if now.Sub(h.lastSeen) > maxAge {
r.forget(id)
}
}
r.entries = slices.DeleteFunc(r.entries, func(e activityEvent) bool {
return now.Sub(e.receivedAt) > maxAge
})
}

// activityEventsSnapshot is the events and drop counts of a set of hives.
Expand Down
12 changes: 12 additions & 0 deletions src/pkg/hub/activity_events_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,18 @@ func TestActivityEventRing_FloodCannotEvictOtherHive(t *testing.T) {
}
}

func TestActivityEventRing_OldEventExpiresWhileHiveKeepsHeartbeating(t *testing.T) {
var st activityStore
st.record("hive-b", withEvents(activityDigest("o/b", 1), activityEv(1, hubactivity.EventStage, "o/b", 1)), activityT0)
for h := time.Duration(1); h <= 30; h++ {
st.record("hive-b", activityDigest("o/b", 1), activityT0.Add(h*time.Hour))
}
b := st.eventsSnapshot([]string{"hive-b"})
if len(b.events) != 0 || b.dropped["hive-b"] != 0 {
t.Fatalf("hive-b has %d events, dropped %d; want its old event aged out without counting as a drop", len(b.events), b.dropped["hive-b"])
}
}

func TestActivityEventRing_DropsMalformedEvents(t *testing.T) {
var st activityStore
st.record("hive-a", withEvents(activityDigest("o/a", 1),
Expand Down
Loading