diff --git a/changelog.d/fixed-memory-growth-dashboard-stage-tracker.md b/changelog.d/fixed-memory-growth-dashboard-stage-tracker.md new file mode 100644 index 0000000000..5127bcf15c --- /dev/null +++ b/changelog.d/fixed-memory-growth-dashboard-stage-tracker.md @@ -0,0 +1 @@ +- Stopped the unbounded memory, thread and connection growth in the hive process that OOM-killed hosted hives ([#14681](https://github.com/hivecommons/hive/issues/14681)): `/api/stage-tracker` held its cache mutex across the whole tracker build (one review-evidence lookup per open PR, each a glob of the repo's evidence directory on NFS — tens of seconds), and every webhook delivery and every status publish invalidated that cache through the same mutex, so webhook handlers, the governor's publish and every dashboard poll queued behind back-to-back rebuilds that never drained. Invalidation is now a flag flip that never waits, at most one rebuild runs at a time outside the lock, concurrent polls get the previous body as `source: stale`, a poll whose client went away stops waiting, and the evidence-directory sibling listing is memoised per repo directory for 60 s instead of re-read per PR. diff --git a/src/docs/api-reference.md b/src/docs/api-reference.md index 2396b271ee..20ce96a058 100644 --- a/src/docs/api-reference.md +++ b/src/docs/api-reference.md @@ -13,17 +13,17 @@ History point limits, timestamp cutoffs, incremental cost reads and ETag/304 beh | `GET` | `/api/version` | Dashboard auth/session | Build/version metadata; includes `upgradeMarker` (`target`, `current`, `attempts`, `maxAttempts`, `failed`, `requestedAt`, `lastError`) while a self-upgrade is in flight or has failed ([#6765](https://github.com/hivecommons/hive/issues/6765)), the hub-delivered `upgradePolicy` (`hub_managed`, `spoke_managed`, `schedule`, `schedule_hour`, `schedule_timezone`, `schedule_weekday`, `paused`, `branch`, `channel`, `target_sha`, `target_resolved`, `armed_target`, and `next_update_at` — the hub's RFC3339 ETA for the next promotion into the `stable` channel, omitted when unknown ([#10256](https://github.com/hivecommons/hive/issues/10256))) when available, and an `autoUpdate` object (`enabled`, `state` — one of `disabled`/`up_to_date`/`behind`/`retrying`/`failed`/`unknown` — `healthy`, `period`, `targetBranch`, `targetCommit`, `currentCommit`, `commitsBehind`, `lastAttemptAt`, `lastError`, `detail`, and `nextUpdateAt` — the hub's RFC3339 ETA for the next update, omitted when unknown, disabled or paused ([#10257](https://github.com/hivecommons/hive/issues/10257))) that never reports a failed or unknown update as healthy ([#6962](https://github.com/hivecommons/hive/issues/6962), [#6963](https://github.com/hivecommons/hive/issues/6963)) | `pkg/dashboard/api.go:42` | | `GET` | `/api/version/release-notes` | Dashboard auth/session | Changelog gap between two revisions: `?from=&to=` returns `{from:{sha,version?}, to:{sha,ref,version?}, sections:[{version,date,categories:{added,changed,fixed,security,deprecated}}], unreleased?, truncated, source, error?}` — the `CHANGELOG.md` release sections present at `to` and absent at `from`, newest first, plus (when `to` is not a release tag) the `changelog.d/` fragments new at `to` under an `Unreleased (in this build)` section. Fetch failures return `source: "unavailable"` with `error` rather than an error status; results are cached per (from, to) for an hour ([#11065](https://github.com/hivecommons/hive/issues/11065)) | `pkg/dashboard/api.go:43` | | `POST` | `/api/release-channel` | Owner only | Hosted spoke self-service release-channel selector; relays `stable`/`candidate`/`edge` to the hub's existing switch-branch endpoint with the spoke dashboard-token proof, and reports the requested channel as pending until the Deployment image lands | `pkg/dashboard/api.go:64` | -| `GET` | `/api/health` | Public | Basic health probe | `pkg/dashboard/server.go:1489` | -| `GET` | `/api/health/deep` | Public | Deep health probe | `pkg/dashboard/server.go:1490` | -| `GET` | `/api/livez` | Public | Kubernetes liveness probe | `pkg/dashboard/server.go:1491` | -| `GET` | `/metrics` | Registered only when `HIVE_METRICS_ENABLED`; requires `Authorization: ****** (403 if the token is unset) | Prometheus metrics | `pkg/dashboard/server.go:1497` | -| `GET` | `/api/status` | Dashboard auth/session | Dashboard aggregate status; `labelPresets[]` (`{key,label,description,labels[],states[]}`) lists the built-in label-filter presets for the Projects cards, led by `needs-human` whose `labels[]` are the effective hard-suppress labels plus `needs-signal`, the hold labels, `stage/071-needs-author` and the reporter-trust awaiting label (#12172); accepts `?fields=a,b` for top-level selection and `?omit=repos` to drop heavy sections, and honors `Accept-Encoding: gzip`; `agents[]` includes busy-pane visibility fields (`kicksUndeliverable`, `busySince`, `lastTranscriptActivity`, `condition`, `conditionMessage`) when a Working pane cannot accept kicks; a working agent also carries `agents[].run` — the live per-run line (#14576): `{since, elapsedS, costUSD, tokens, priced, tests, filesChanged}`, the estimated spend/tokens since the kick was delivered (`lastKickAt`, falling back to `busySince`) (delta of the token collector's per-agent cumulative figure; `priced:false` when the agent has none yet), the last visible test-runner result (`pass`/`fail`/empty) and the last `N files changed` git summary (`-1` when none) parsed from the captured pane; top-level `fleetRun` `{running, hourUSD, hourPriced}` is the governor strip's `running N · $X this hour` figure; `sla` (fleet) and `repos[].sla` (per repo) carry the Delivery SLA block derived from the lifecycle timeline with no extra GitHub calls: `windows.{7d,30d}.{open_to_triage,triage_to_start,start_to_pr,pr_to_merge,open_to_merge}` as `{p50_h,p95_h,n}` (hours, nearest-rank), `history[]` of up to 30 daily buckets `{day,start_to_pr_p50,pr_to_merge_p50,open_to_merge_p50,n_started,n_pr,n_merged}` persisted under `/data/sla-history.json`, `stale_in_progress[]` (`{number,repo,started_at,age_h,agent}` — issues carrying `hive/in-progress` longer than the agent claim TTL with no linked PR), `stale_count`, `merged_7d`, `claim_ttl_s` and, on the fleet block, the effective `targets` (`start_to_pr_h`, `pr_to_merge_h`, `open_to_merge_h`); `staleSince` and `repos[].countsStale` / `countsStaleSince` mark totals served from last-good figures — the previous enumeration (#13382) or the ForgeState cache's last-good counts (#13376) | `pkg/dashboard/server.go:1502` | -| `GET` | `/api/status/summary` | Dashboard auth/session | Compact agent/governor/budget status summary for pollers; honors `Accept-Encoding: gzip` | `pkg/dashboard/server.go:1503` | -| `GET` | `/api/overview/issues.csv` | Dashboard auth/session | Cached Overview issue-band export as browser-compatible CSV; filters `band`, `repo`/`repos`, `stale`, and `held` are ANDed. | `pkg/dashboard/server.go:1504` | -| `GET` | `/api/overview/issues.json` | Dashboard auth/session | Cached Overview issue-band export as typed JSON with `bands[]` metadata and row fields matching the CSV columns. | `pkg/dashboard/server.go:1505` | -| `GET` | `/api/overview/prs.csv` | Dashboard auth/session | Cached Overview PR-band export as browser-compatible CSV; filters `band`, `repo`/`repos`, `stale`, and `held` are ANDed. | `pkg/dashboard/server.go:1506` | -| `GET` | `/api/overview/prs.json` | Dashboard auth/session | Cached Overview PR-band export as typed JSON with `bands[]` metadata and row fields matching the CSV columns. | `pkg/dashboard/server.go:1507` | -| `GET` | `/api/events` | Dashboard auth/session | Server-sent event stream | `pkg/dashboard/server.go:1508` | +| `GET` | `/api/health` | Public | Basic health probe | `pkg/dashboard/server.go:1498` | +| `GET` | `/api/health/deep` | Public | Deep health probe | `pkg/dashboard/server.go:1499` | +| `GET` | `/api/livez` | Public | Kubernetes liveness probe | `pkg/dashboard/server.go:1500` | +| `GET` | `/metrics` | Registered only when `HIVE_METRICS_ENABLED`; requires `Authorization: ****** (403 if the token is unset) | Prometheus metrics | `pkg/dashboard/server.go:1506` | +| `GET` | `/api/status` | Dashboard auth/session | Dashboard aggregate status; `labelPresets[]` (`{key,label,description,labels[],states[]}`) lists the built-in label-filter presets for the Projects cards, led by `needs-human` whose `labels[]` are the effective hard-suppress labels plus `needs-signal`, the hold labels, `stage/071-needs-author` and the reporter-trust awaiting label (#12172); accepts `?fields=a,b` for top-level selection and `?omit=repos` to drop heavy sections, and honors `Accept-Encoding: gzip`; `agents[]` includes busy-pane visibility fields (`kicksUndeliverable`, `busySince`, `lastTranscriptActivity`, `condition`, `conditionMessage`) when a Working pane cannot accept kicks; a working agent also carries `agents[].run` — the live per-run line (#14576): `{since, elapsedS, costUSD, tokens, priced, tests, filesChanged}`, the estimated spend/tokens since the kick was delivered (`lastKickAt`, falling back to `busySince`) (delta of the token collector's per-agent cumulative figure; `priced:false` when the agent has none yet), the last visible test-runner result (`pass`/`fail`/empty) and the last `N files changed` git summary (`-1` when none) parsed from the captured pane; top-level `fleetRun` `{running, hourUSD, hourPriced}` is the governor strip's `running N · $X this hour` figure; `sla` (fleet) and `repos[].sla` (per repo) carry the Delivery SLA block derived from the lifecycle timeline with no extra GitHub calls: `windows.{7d,30d}.{open_to_triage,triage_to_start,start_to_pr,pr_to_merge,open_to_merge}` as `{p50_h,p95_h,n}` (hours, nearest-rank), `history[]` of up to 30 daily buckets `{day,start_to_pr_p50,pr_to_merge_p50,open_to_merge_p50,n_started,n_pr,n_merged}` persisted under `/data/sla-history.json`, `stale_in_progress[]` (`{number,repo,started_at,age_h,agent}` — issues carrying `hive/in-progress` longer than the agent claim TTL with no linked PR), `stale_count`, `merged_7d`, `claim_ttl_s` and, on the fleet block, the effective `targets` (`start_to_pr_h`, `pr_to_merge_h`, `open_to_merge_h`); `staleSince` and `repos[].countsStale` / `countsStaleSince` mark totals served from last-good figures — the previous enumeration (#13382) or the ForgeState cache's last-good counts (#13376) | `pkg/dashboard/server.go:1511` | +| `GET` | `/api/status/summary` | Dashboard auth/session | Compact agent/governor/budget status summary for pollers; honors `Accept-Encoding: gzip` | `pkg/dashboard/server.go:1512` | +| `GET` | `/api/overview/issues.csv` | Dashboard auth/session | Cached Overview issue-band export as browser-compatible CSV; filters `band`, `repo`/`repos`, `stale`, and `held` are ANDed. | `pkg/dashboard/server.go:1513` | +| `GET` | `/api/overview/issues.json` | Dashboard auth/session | Cached Overview issue-band export as typed JSON with `bands[]` metadata and row fields matching the CSV columns. | `pkg/dashboard/server.go:1514` | +| `GET` | `/api/overview/prs.csv` | Dashboard auth/session | Cached Overview PR-band export as browser-compatible CSV; filters `band`, `repo`/`repos`, `stale`, and `held` are ANDed. | `pkg/dashboard/server.go:1515` | +| `GET` | `/api/overview/prs.json` | Dashboard auth/session | Cached Overview PR-band export as typed JSON with `bands[]` metadata and row fields matching the CSV columns. | `pkg/dashboard/server.go:1516` | +| `GET` | `/api/events` | Dashboard auth/session | Server-sent event stream | `pkg/dashboard/server.go:1517` | | `GET` | `/api/swarm` | Dashboard auth/session | Current swarm status: display name, duration, active repo, expiry, persisted prep metrics when available, and idle-unlock fields. | `pkg/dashboard/swarm.go:310` | | `POST` | `/api/swarm` | Owner only | Start a 24h repo swarm for a configured repo, returning 409 when another swarm is active and 423 when a previous swarm keeps the next swarm locked until enough agents are idle unless body `{"repo":"owner/name","force":true}` is used; saves the active record before repo prep, persists prep metrics, and announces the start to Discord when configured. | `pkg/dashboard/swarm.go:311` | | `DELETE` | `/api/swarm` | Owner only | End the active swarm, score issues closed and PRs merged in the window, append it to history, and announce the result to Discord when configured. | `pkg/dashboard/swarm.go:312` | @@ -66,7 +66,7 @@ History point limits, timestamp cutoffs, incremental cost reads and ETag/304 beh | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| | `GET` | `/api/style` | Public | Sanitized custom dashboard CSS | `pkg/dashboard/api.go:44` | -| `GET` | `/branding/custom.css` | Dashboard auth/session | Operator branding stylesheet override, read per request (see [branding](branding.md)) | `pkg/dashboard/server.go:1569` | +| `GET` | `/branding/custom.css` | Dashboard auth/session | Operator branding stylesheet override, read per request (see [branding](branding.md)) | `pkg/dashboard/server.go:1578` | | `GET` | `/api/snapshot/frame-ancestors` | Public | Snapshot framing allowlist | `pkg/dashboard/api.go:77` | | `GET` | `/api/snapshot` | Public | Snapshot data | `pkg/dashboard/api.go:78` | | `GET` | `/snapshot` | Public | Public read-only snapshot page | `pkg/dashboard/api.go:79` | @@ -94,9 +94,9 @@ History point limits, timestamp cutoffs, incremental cost reads and ETag/304 beh | `GET` | `/api/openrouter/models` | Dashboard auth/session | Open Router Models | `pkg/dashboard/openrouter.go:47` | | `GET` | `/api/openrouter/credit` | Dashboard auth/session | Open Router Credit | `pkg/dashboard/openrouter.go:48` | | `GET` | `/openrouter/callback` | Public | Open Router Callback | `pkg/dashboard/openrouter.go:49` | -| `POST` | `/api/github-app/recheck` | Dashboard auth/session | GitHub App Recheck | `pkg/dashboard/server.go:1509` | -| `POST` | `/api/github-app/install-clicked` | Dashboard auth/session | GitHub App Install Clicked | `pkg/dashboard/server.go:1510` | -| `GET` | `/gh-setup` | Public | GitHub App Setup Callback | `pkg/dashboard/server.go:1511` | +| `POST` | `/api/github-app/recheck` | Dashboard auth/session | GitHub App Recheck | `pkg/dashboard/server.go:1518` | +| `POST` | `/api/github-app/install-clicked` | Dashboard auth/session | GitHub App Install Clicked | `pkg/dashboard/server.go:1519` | +| `GET` | `/gh-setup` | Public | GitHub App Setup Callback | `pkg/dashboard/server.go:1520` | ## Configuration @@ -569,7 +569,7 @@ always resolved server-side from the validated token. | `GET` | `/api/budget-ignore` | Dashboard auth/session | Budget Ignore Get | `pkg/dashboard/api.go:170` | | `POST` | `/api/budget-ignore` | Owner only | Budget Ignore Set | `pkg/dashboard/api.go:171` | | `GET` | `/api/summaries` | Dashboard auth/session | Summaries | `pkg/dashboard/api.go:183` | -| `GET` | `/api/stage-tracker` | Dashboard auth/session | Stage Tracker: every open issue and pull request of the watched repos for the Hive activity monitor — actionable, held, claimed, in-progress, covered-by-PR, filtered (`needs-*`), draft and exempt-labelled items alike, each with `stage`, `stage_since`, `labels`, `created_at`, `author_kind`, `modifiers`, (PRs) `sentinel_alert` / `sentinel_findings` / `sentinel_reason` when the sentinel flagged the PR, and (issues) `linked_prs`; items with no stage yet report stage `0` "Open"; `needs_human_labels` is the "a person must act" label set behind the monitor's Needs human facet (the same list as `/api/status.labelPresets[needs-human]`); `recently_merged` lists the PRs (`repo`, `number`, `merged_at`) whose `pr_merged` audit entry is under 30 minutes old, so a client diffing two payloads can tell a merge from a close. Served from a server-side cache rebuilt at most every `dashboard.stage_tracker.cache_ttl_s` (default `governor.eval_interval_s`) from the spoke's existing snapshot — a request never calls GitHub. Carries `generated_at`, `source` (`cache` or `fresh`), `cache_ttl_s`, an `ETag` and `Cache-Control: private, max-age=`; `If-None-Match` yields 304. Each hit is recorded under REST caller `dashboard:stage-tracker` so `/api/gh-rate-limits` shows it charged nothing. | `pkg/dashboard/api.go:184` | +| `GET` | `/api/stage-tracker` | Dashboard auth/session | Stage Tracker: every open issue and pull request of the watched repos for the Hive activity monitor — actionable, held, claimed, in-progress, covered-by-PR, filtered (`needs-*`), draft and exempt-labelled items alike, each with `stage`, `stage_since`, `labels`, `created_at`, `author_kind`, `modifiers`, (PRs) `sentinel_alert` / `sentinel_findings` / `sentinel_reason` when the sentinel flagged the PR, and (issues) `linked_prs`; items with no stage yet report stage `0` "Open"; `needs_human_labels` is the "a person must act" label set behind the monitor's Needs human facet (the same list as `/api/status.labelPresets[needs-human]`); `recently_merged` lists the PRs (`repo`, `number`, `merged_at`) whose `pr_merged` audit entry is under 30 minutes old, so a client diffing two payloads can tell a merge from a close. Served from a server-side cache rebuilt at most every `dashboard.stage_tracker.cache_ttl_s` (default `governor.eval_interval_s`) from the spoke's existing snapshot — a request never calls GitHub. Carries `generated_at`, `source` (`cache`, `fresh`, or `stale` — the previous body served at once while one rebuild runs; at most one rebuild is ever in flight, and a status publish or webhook only marks the cache stale, it never waits), `cache_ttl_s`, an `ETag` and `Cache-Control: private, max-age=` (`0` for `stale`); `If-None-Match` yields 304. A request that arrives before any body exists waits for the in-flight build and answers 503 if its own connection closes first. Each hit is recorded under REST caller `dashboard:stage-tracker` so `/api/gh-rate-limits` shows it charged nothing. | `pkg/dashboard/api.go:184` | | `POST` | `/api/prs/{owner}/{repo}/{number}/queue-automerge` | Merger/owner role | Queue PRAuto Merge | `pkg/dashboard/api.go:185` | | `GET` | `/api/inference/models/{backend}` | Dashboard auth/session | Inference Models | `pkg/dashboard/api.go:355` | | `GET` | `/api/hive-id` | Dashboard auth/session | Hive IDGet | `pkg/dashboard/api.go:417` | @@ -593,7 +593,7 @@ always resolved server-side from the validated token. | `POST` | `/api/hives/{id}/heartbeat` | Dashboard auth/session | Hives Heartbeat | `pkg/dashboard/api_contribute.go:275` | | `DELETE` | `/api/hives/{id}` | Owner only | Hives Delete | `pkg/dashboard/api_contribute.go:276` | | `POST` | `/api/hives/onboard` | Dashboard auth/session | Hives Onboard | `pkg/dashboard/api_contribute.go:277` | -| `GET` | `/sso` | Public | SSO | `pkg/dashboard/server.go:1516` | +| `GET` | `/sso` | Public | SSO | `pkg/dashboard/server.go:1525` | ## Hub SaaS diff --git a/src/docs/operator-reference.md b/src/docs/operator-reference.md index e1acb43881..269e3fab9c 100644 --- a/src/docs/operator-reference.md +++ b/src/docs/operator-reference.md @@ -1222,7 +1222,14 @@ v5 and v6 spokes) tracking one repo all stay on webhook mode. - Every status publish (cadence or wake) invalidates the `/api/stage-tracker` cache and sends a `stage-tracker` event on `/api/events`, so the dashboard's activity monitor refetches at once instead of waiting out - `dashboard.stage_tracker.cache_ttl_s` plus its poll. + `dashboard.stage_tracker.cache_ttl_s` plus its poll. Invalidation only + marks the cached body stale: the publish, and every webhook below, returns + at once even while a tracker rebuild is running. The next poll starts + **one** rebuild outside any lock; polls that land during it get the + previous body with `source: stale` and `max-age=0` rather than queueing, + so a slow build (one sentinel evidence read per open PR, on NFS in hosted + hives) can never back up webhook handlers, the governor's publish, or an + unbounded line of dashboard requests (hivecommons/hive#14681). - **The activity monitor row moves without waiting for a cycle.** A delivery that names a PR or issue this hive tracks (`pull_request*`, `check_suite`/`check_run`, `status`, `push` to a PR branch, `issue_comment`, diff --git a/src/pkg/dashboard/server.go b/src/pkg/dashboard/server.go index 10a752bba3..3a013cdc4d 100644 --- a/src/pkg/dashboard/server.go +++ b/src/pkg/dashboard/server.go @@ -121,6 +121,15 @@ type Server struct { // the status snapshot, served until its TTL lapses (stage_tracker.go). stageTrackerCache stageTrackerCache stageTrackerCacheMu sync.Mutex + // stageTrackerBuild is the one rebuild in flight, nil when none; it is + // guarded by stageTrackerCacheMu, which is never held across the build. + stageTrackerBuild *stageTrackerBuild + // stageTrackerGen counts invalidations so a build can tell whether the + // status it rendered was superseded while it ran. + stageTrackerGen uint64 + // stageTrackerBuildFn replaces buildStageTracker in tests that need a + // build they can hold open; nil in production. + stageTrackerBuildFn func(status *StatusPayload, store *stagelabels.Store, now time.Time) StageTrackerResponse // evidenceHB is the background-refreshed evidence aggregate the hub // heartbeat reads (evidence_heartbeat.go); zero value is ready to use. evidenceHB evidenceHeartbeatCache diff --git a/src/pkg/dashboard/stage_tracker.go b/src/pkg/dashboard/stage_tracker.go index 2d5483ad0b..6fde303e12 100644 --- a/src/pkg/dashboard/stage_tracker.go +++ b/src/pkg/dashboard/stage_tracker.go @@ -2,6 +2,7 @@ package dashboard import ( "bytes" + "context" "encoding/json" "fmt" "net/http" @@ -1545,8 +1546,29 @@ type stageTrackerCache struct { body []byte etag string generatedAt time.Time + // stale is set by invalidateStageTrackerCache: the body is still served + // while the next request rebuilds, so an invalidation never blocks and a + // reader never waits behind a build it did not start. + stale bool } +// stageTrackerBuild is the single in-flight /api/stage-tracker rebuild. done +// is closed when the cache has been updated (or the build failed). +type stageTrackerBuild struct { + done chan struct{} +} + +// stageTrackerSource values for the X-Hive-Stage-Tracker-Source header and +// the payload's source field. +const ( + stageTrackerSourceFreshName = "fresh" + stageTrackerSourceCacheName = "cache" + // stageTrackerSourceStaleName marks a body served from the cache while a + // rebuild is in flight or after an invalidation: still the last complete + // tracker, just older than the status it will be rebuilt from. + stageTrackerSourceStaleName = "stale" +) + func (s *Server) stageTrackerCacheTTL() time.Duration { if s != nil && s.deps != nil && s.deps.Config != nil { return s.deps.Config.StageTrackerCacheTTL() @@ -1555,22 +1577,94 @@ func (s *Server) stageTrackerCacheTTL() time.Duration { } // stageTrackerPayload returns the cached body when it is younger than the -// TTL, else builds, encodes and caches a new one. fresh reports a rebuild. +// TTL and not invalidated, else builds, encodes and caches a new one. fresh +// reports a rebuild by this call. +// +// The build runs OUTSIDE stageTrackerCacheMu and at most one runs at a time: +// the mutex only guards the cached fields, so invalidateStageTrackerCache +// (every webhook, every status publish) returns at once instead of queueing +// behind a build that walks the evidence tree on NFS. Requests that arrive +// while a build is in flight are served the previous body as "stale"; only a +// request with no body to fall back on waits, and only until the in-flight +// build finishes or its own context ends. Before this, the mutex was held +// across the build, so every webhook handler, the governor's publish and +// every tracker poll queued behind it in FIFO order — and because each +// publish and webhook re-invalidated the cache, the queue never drained +// (hivecommons/hive#14681: unbounded blocked handlers, connections and +// threads). func (s *Server) stageTrackerPayload(status *StatusPayload, now time.Time) (body []byte, etag string, generatedAt time.Time, fresh bool) { + var source string + body, etag, generatedAt, source = s.stageTrackerPayloadSource(context.Background(), status, now) + return body, etag, generatedAt, source == stageTrackerSourceFreshName +} + +// stageTrackerPayloadSource is stageTrackerPayload reporting where the body +// came from (stageTrackerSource*Name; "" with a nil body when the build +// failed or ctx ended while waiting on an in-flight build). +func (s *Server) stageTrackerPayloadSource(ctx context.Context, status *StatusPayload, now time.Time) (body []byte, etag string, generatedAt time.Time, source string) { ttl := s.stageTrackerCacheTTL() s.stageTrackerCacheMu.Lock() - defer s.stageTrackerCacheMu.Unlock() s.stageTrackerServed = true - c := &s.stageTrackerCache - if len(c.body) > 0 && now.Sub(c.generatedAt) < ttl { - return c.body, c.etag, c.generatedAt, false + c := s.stageTrackerCache + if len(c.body) > 0 && !c.stale && now.Sub(c.generatedAt) < ttl { + s.stageTrackerCacheMu.Unlock() + return c.body, c.etag, c.generatedAt, stageTrackerSourceCacheName + } + if inflight := s.stageTrackerBuild; inflight != nil { + s.stageTrackerCacheMu.Unlock() + if len(c.body) > 0 { + return c.body, c.etag, c.generatedAt, stageTrackerSourceStaleName + } + select { + case <-inflight.done: + case <-ctx.Done(): + return nil, "", now, "" + } + s.stageTrackerCacheMu.Lock() + c = s.stageTrackerCache + s.stageTrackerCacheMu.Unlock() + if len(c.body) == 0 { + return nil, "", now, "" + } + return c.body, c.etag, c.generatedAt, stageTrackerSourceCacheName + } + inflight := &stageTrackerBuild{done: make(chan struct{})} + s.stageTrackerBuild = inflight + gen := s.stageTrackerGen + s.stageTrackerCacheMu.Unlock() + + built, ok := s.encodeStageTracker(status, now, ttl) + + s.stageTrackerCacheMu.Lock() + if ok { + // An invalidation that landed during the build means newer status + // exists; keep serving this body but let the next request rebuild. + built.stale = s.stageTrackerGen != gen + s.stageTrackerCache = built + } + s.stageTrackerBuild = nil + close(inflight.done) + s.stageTrackerCacheMu.Unlock() + if !ok { + return nil, "", now, "" } - resp := s.buildStageTracker(status, s.stageStoreOrShared(), now) + return built.body, built.etag, built.generatedAt, stageTrackerSourceFreshName +} + +// encodeStageTracker builds and encodes the tracker for status. It holds no +// server lock: this is the slow part (one evidence read per PR, on NFS in +// hosted hives). +func (s *Server) encodeStageTracker(status *StatusPayload, now time.Time, ttl time.Duration) (stageTrackerCache, bool) { + build := s.buildStageTracker + if s.stageTrackerBuildFn != nil { + build = s.stageTrackerBuildFn + } + resp := build(status, s.stageStoreOrShared(), now) resp.CacheTTLSeconds = int64(ttl / time.Second) // The encoded body carries source=cache: that is what every later hit // serves verbatim, and what the ETag validates. The building request // alone rewrites the marker to fresh on its way out (handleStageTracker). - resp.Source = "cache" + resp.Source = stageTrackerSourceCacheName // The ETag hashes the body with generated_at zeroed: the timestamp moves // on every rebuild, so including it would turn an unchanged tracker into // a 200 each time (the client revalidates with this ETag and gets a 304). @@ -1579,26 +1673,29 @@ func (s *Server) stageTrackerPayload(status *StatusPayload, now time.Time) (body hashResp.GeneratedAt = time.Time{} if err := json.NewEncoder(&hashBuf).Encode(hashResp); err != nil { slogWarnOverviewEncode(err) - return nil, "", now, true + return stageTrackerCache{}, false } var buf bytes.Buffer if err := json.NewEncoder(&buf).Encode(resp); err != nil { slogWarnOverviewEncode(err) - return nil, "", now, true + return stageTrackerCache{}, false } - c.body, c.etag, c.generatedAt = buf.Bytes(), webstatic.ETagFor(hashBuf.Bytes()), now - return c.body, c.etag, c.generatedAt, true + return stageTrackerCache{body: buf.Bytes(), etag: webstatic.ETagFor(hashBuf.Bytes()), generatedAt: now}, true } -// invalidateStageTrackerCache forgets the memoised payload. UpdateStatusIfFresh -// calls it on every status publish so tracker freshness follows the eval -// cycle rather than the cache TTL alone. It reports whether the tracker has -// ever been served: only then is an SSE nudge worth sending, which also keeps -// the /api/events stream free of extra frames for clients that never asked -// for the tracker. +// invalidateStageTrackerCache marks the memoised payload stale. +// UpdateStatusIfFresh calls it on every status publish, and every webhook +// re-read that changed a row calls it too, so tracker freshness follows the +// eval cycle and the webhook stream rather than the cache TTL alone. It only +// flips a flag under the cache mutex — it never waits for a build in +// progress — so callers on the webhook and publish paths return at once. It +// reports whether the tracker has ever been served: only then is an SSE +// nudge worth sending, which also keeps the /api/events stream free of extra +// frames for clients that never asked for the tracker. func (s *Server) invalidateStageTrackerCache() (served bool) { s.stageTrackerCacheMu.Lock() - s.stageTrackerCache = stageTrackerCache{} + s.stageTrackerCache.stale = true + s.stageTrackerGen++ served = s.stageTrackerServed s.stageTrackerCacheMu.Unlock() return served @@ -1613,6 +1710,7 @@ func stageTrackerSSEFrame(seq uint64) string { var stageTrackerSourceCache = []byte(`"source":"cache"`) var stageTrackerSourceFresh = []byte(`"source":"fresh"`) +var stageTrackerSourceStale = []byte(`"source":"stale"`) // handleStageTracker serves GET /api/stage-tracker from the memoised payload. func (s *Server) handleStageTracker(w http.ResponseWriter, r *http.Request) { @@ -1625,18 +1723,22 @@ func (s *Server) handleStageTracker(w http.ResponseWriter, r *http.Request) { return } now := time.Now().UTC() - body, etag, generatedAt, fresh := s.stageTrackerPayload(status, now) + body, etag, generatedAt, source := s.stageTrackerPayloadSource(r.Context(), status, now) if body == nil { + if r.Context().Err() != nil { + jsonError(w, "stage tracker build still in progress", http.StatusServiceUnavailable) + return + } jsonError(w, "stage tracker unavailable", http.StatusInternalServerError) return } remaining := s.stageTrackerCacheTTL() - now.Sub(generatedAt) - if remaining < 0 { + if remaining < 0 || source == stageTrackerSourceStaleName { remaining = 0 } w.Header().Set("ETag", etag) w.Header().Set("Cache-Control", "private, max-age="+strconv.FormatInt(int64(remaining/time.Second), 10)) - w.Header().Set("X-Hive-Stage-Tracker-Source", map[bool]string{true: "fresh", false: "cache"}[fresh]) + w.Header().Set("X-Hive-Stage-Tracker-Source", source) w.Header().Set("X-Hive-Stage-Tracker-Generated-At", generatedAt.UTC().Format(time.RFC3339Nano)) w.Header().Set("Last-Modified", generatedAt.UTC().Format(http.TimeFormat)) if match := r.Header.Get("If-None-Match"); match != "" && strings.Contains(match, etag) { @@ -1645,8 +1747,11 @@ func (s *Server) handleStageTracker(w http.ResponseWriter, r *http.Request) { return } github.RecordLocalServe(stageTrackerRESTCaller, r.URL.Path, http.StatusOK) - if fresh { + switch source { + case stageTrackerSourceFreshName: body = bytes.Replace(body, stageTrackerSourceCache, stageTrackerSourceFresh, 1) + case stageTrackerSourceStaleName: + body = bytes.Replace(body, stageTrackerSourceCache, stageTrackerSourceStale, 1) } if _, err := w.Write(body); err != nil { slogWarnOverviewEncode(err) diff --git a/src/pkg/dashboard/stage_tracker_convoy_test.go b/src/pkg/dashboard/stage_tracker_convoy_test.go new file mode 100644 index 0000000000..c96bd88a9e --- /dev/null +++ b/src/pkg/dashboard/stage_tracker_convoy_test.go @@ -0,0 +1,254 @@ +package dashboard + +import ( + "context" + "net/http" + "net/http/httptest" + "runtime" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/hivecommons/hive/internal/testutil" + "github.com/hivecommons/hive/pkg/stagelabels" +) + +// heldStageTrackerBuild is a tracker build the test holds open: each call +// blocks on release and counts itself, standing in for the 40s evidence walk +// on NFS that hivecommons/hive#14681 measured. +type heldStageTrackerBuild struct { + release chan struct{} + started chan struct{} // one send per build entered + builds atomic.Int64 +} + +func newHeldStageTrackerBuild() *heldStageTrackerBuild { + return &heldStageTrackerBuild{release: make(chan struct{}), started: make(chan struct{}, 64)} +} + +func (h *heldStageTrackerBuild) build(status *StatusPayload, _ *stagelabels.Store, now time.Time) StageTrackerResponse { + h.builds.Add(1) + h.started <- struct{}{} + <-h.release + return StageTrackerResponse{GeneratedAt: now, Items: []StageTrackerItem{{Repo: "acme/console", Number: int(h.builds.Load())}}} +} + +func (h *heldStageTrackerBuild) waitStarted(t *testing.T) { + t.Helper() + select { + case <-h.started: + case <-time.After(5 * time.Second): + t.Fatal("no tracker build started") + } +} + +func convoyServer(t *testing.T, h *heldStageTrackerBuild) *Server { + t.Helper() + s := newTestServer() + deps := testDeps(t) + deps.Config.Dashboard.StageTracker.CacheTTLS = 3600 + s.deps = deps + s.stageTrackerBuildFn = h.build + s.UpdateStatus(minimalPayload()) + return s +} + +func getStageTracker(ctx context.Context, s *Server) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodGet, "/api/stage-tracker", nil).WithContext(ctx) + rr := httptest.NewRecorder() + s.handleStageTracker(rr, req) + return rr +} + +// TestStageTrackerInvalidationNeverWaitsForABuild pins the fix for #14681: +// the webhook and status-publish paths invalidate the tracker cache, and +// they must return while a build is in flight. Before, the invalidator took +// the mutex the builder held for the whole build, so every webhook handler +// and the governor's publish queued behind a 40s NFS walk. +func TestStageTrackerInvalidationNeverWaitsForABuild(t *testing.T) { + h := newHeldStageTrackerBuild() + s := convoyServer(t, h) + + first := make(chan *httptest.ResponseRecorder, 1) + go func() { first <- getStageTracker(context.Background(), s) }() + h.waitStarted(t) + + invalidated := make(chan bool, 1) + go func() { invalidated <- s.invalidateStageTrackerCache() }() + select { + case served := <-invalidated: + if !served { + t.Fatal("invalidate must report the tracker as served once a build began") + } + case <-time.After(5 * time.Second): + t.Fatal("invalidateStageTrackerCache blocked behind the in-flight build") + } + // The webhook path goes through the same door. + done := make(chan struct{}) + go func() { s.invalidateStageTrackerForWebhook(); close(done) }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("invalidateStageTrackerForWebhook blocked behind the in-flight build") + } + + close(h.release) + rr := <-first + if rr.Code != http.StatusOK || rr.Header().Get("X-Hive-Stage-Tracker-Source") != "fresh" { + t.Fatalf("building request: %d src=%q", rr.Code, rr.Header().Get("X-Hive-Stage-Tracker-Source")) + } + if n := h.builds.Load(); n != 1 { + t.Fatalf("builds = %d, want 1", n) + } +} + +// TestStageTrackerBuildsAreSingleFlightAndServeStale: N concurrent polls +// during a rebuild cost one build and return at once with the previous body +// marked stale, so the number of goroutines parked in the handler cannot +// grow with the poll rate — the pile-up the production hive leaked under. +func TestStageTrackerBuildsAreSingleFlightAndServeStale(t *testing.T) { + h := newHeldStageTrackerBuild() + s := convoyServer(t, h) + + // Seed the cache with one complete build. + seed := make(chan *httptest.ResponseRecorder, 1) + go func() { seed <- getStageTracker(context.Background(), s) }() + h.waitStarted(t) + h.release <- struct{}{} + if rr := <-seed; rr.Code != http.StatusOK { + t.Fatalf("seed: %d", rr.Code) + } + etag := func(rr *httptest.ResponseRecorder) string { return rr.Header().Get("ETag") } + seedETag := "" + if rr := getStageTracker(context.Background(), s); rr.Header().Get("X-Hive-Stage-Tracker-Source") != "cache" { + t.Fatalf("second poll must hit the cache, got %q", rr.Header().Get("X-Hive-Stage-Tracker-Source")) + } else { + seedETag = etag(rr) + } + + // Invalidate (a webhook landed); the next poll starts the one rebuild. + s.invalidateStageTrackerCache() + builder := make(chan *httptest.ResponseRecorder, 1) + go func() { builder <- getStageTracker(context.Background(), s) }() + h.waitStarted(t) + + baseline := runtime.NumGoroutine() + const polls = 200 + var wg sync.WaitGroup + codes := make([]int, polls) + sources := make([]string, polls) + for i := 0; i < polls; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + // Each poll is also preceded by a webhook invalidation, as in + // production where ~3 deliveries/s re-stale the cache. + s.invalidateStageTrackerForWebhook() + rr := getStageTracker(context.Background(), s) + codes[i], sources[i] = rr.Code, rr.Header().Get("X-Hive-Stage-Tracker-Source") + }(i) + } + // All polls return while the build is still held open. + finished := make(chan struct{}) + go func() { wg.Wait(); close(finished) }() + select { + case <-finished: + case <-time.After(10 * time.Second): + t.Fatalf("%d polls did not return while a build was in flight", polls) + } + for i := range codes { + if codes[i] != http.StatusOK || sources[i] != "stale" { + t.Fatalf("poll %d: %d src=%q, want 200 stale", i, codes[i], sources[i]) + } + } + if n := h.builds.Load(); n != 2 { + t.Fatalf("builds = %d, want 2 (seed + one rebuild for %d polls)", n, polls) + } + testutil.Eventually(t, 5*time.Second, func() bool { return runtime.NumGoroutine() <= baseline+2 }, + "goroutines after %d stale polls: %d, baseline %d", polls, runtime.NumGoroutine(), baseline) + + // A waiter whose client went away stops waiting. + ctx, cancel := context.WithCancel(context.Background()) + cancel() + // Drop the body so this request has nothing stale to serve and must wait. + s.stageTrackerCacheMu.Lock() + saved := s.stageTrackerCache + s.stageTrackerCache = stageTrackerCache{} + s.stageTrackerCacheMu.Unlock() + if rr := getStageTracker(ctx, s); rr.Code != http.StatusServiceUnavailable { + t.Fatalf("cancelled waiter: %d, want 503", rr.Code) + } + s.stageTrackerCacheMu.Lock() + s.stageTrackerCache = saved + s.stageTrackerCacheMu.Unlock() + + // Release the build: the builder gets fresh; because invalidations + // landed during the build the result is already stale, so the next poll + // serves it as stale and starts build #3. + h.release <- struct{}{} + rr := <-builder + if rr.Code != http.StatusOK || rr.Header().Get("X-Hive-Stage-Tracker-Source") != "fresh" { + t.Fatalf("builder: %d src=%q", rr.Code, rr.Header().Get("X-Hive-Stage-Tracker-Source")) + } + if etag(rr) == seedETag { + t.Fatal("rebuilt body must carry the new ETag") + } + next := make(chan *httptest.ResponseRecorder, 1) + go func() { next <- getStageTracker(context.Background(), s) }() + h.waitStarted(t) + if during := getStageTracker(context.Background(), s); during.Header().Get("X-Hive-Stage-Tracker-Source") != "stale" || etag(during) != etag(rr) { + t.Fatalf("poll during build #3: src=%q etag=%q want stale %q", during.Header().Get("X-Hive-Stage-Tracker-Source"), etag(during), etag(rr)) + } + close(h.release) + if rr := <-next; rr.Header().Get("X-Hive-Stage-Tracker-Source") != "fresh" { + t.Fatalf("build #3: src=%q", rr.Header().Get("X-Hive-Stage-Tracker-Source")) + } + if n := h.builds.Load(); n != 3 { + t.Fatalf("builds = %d, want 3", n) + } + // Quiet now: polls hit the cache and start nothing. + if rr := getStageTracker(context.Background(), s); rr.Header().Get("X-Hive-Stage-Tracker-Source") != "cache" { + t.Fatalf("quiet poll: src=%q", rr.Header().Get("X-Hive-Stage-Tracker-Source")) + } + if n := h.builds.Load(); n != 3 { + t.Fatalf("quiet poll started build %d", n) + } +} + +// TestStageTrackerFirstWaitersJoinTheOneBuild: with no body yet, concurrent +// first polls all wait on the single in-flight build rather than each +// starting one. +func TestStageTrackerFirstWaitersJoinTheOneBuild(t *testing.T) { + h := newHeldStageTrackerBuild() + s := convoyServer(t, h) + const waiters = 50 + results := make(chan *httptest.ResponseRecorder, waiters) + for i := 0; i < waiters; i++ { + go func() { results <- getStageTracker(context.Background(), s) }() + } + h.waitStarted(t) + close(h.release) + fresh, cached := 0, 0 + for i := 0; i < waiters; i++ { + select { + case rr := <-results: + if rr.Code != http.StatusOK { + t.Fatalf("waiter: %d", rr.Code) + } + switch rr.Header().Get("X-Hive-Stage-Tracker-Source") { + case "fresh": + fresh++ + case "cache": + cached++ + default: + t.Fatalf("waiter src=%q", rr.Header().Get("X-Hive-Stage-Tracker-Source")) + } + case <-time.After(10 * time.Second): + t.Fatal("waiters did not drain after the build was released") + } + } + if fresh != 1 || cached != waiters-1 || h.builds.Load() != 1 { + t.Fatalf("fresh=%d cached=%d builds=%d; want 1/%d/1", fresh, cached, h.builds.Load(), waiters-1) + } +} diff --git a/src/pkg/github/review_evidence.go b/src/pkg/github/review_evidence.go index b211fc990e..9e3abc2bb2 100644 --- a/src/pkg/github/review_evidence.go +++ b/src/pkg/github/review_evidence.go @@ -542,7 +542,10 @@ func writeReviewEvidence(path string, b *evidence.Bundle) error { return err } fallback := reviewEvidenceFallbackPath(path) - if ferr := writeEvidenceBundleFile(fallback, data); ferr != nil { + ferr := writeEvidenceBundleFile(fallback, data) + // The sibling may be new: the memoised listing of its parent is stale. + forgetEvidenceSiblings(reviewEvidenceCanonicalDir(filepath.Dir(fallback))) + if ferr != nil { return fmt.Errorf("%w (fallback %s: %v)", err, fallback, ferr) } return nil @@ -594,14 +597,26 @@ func reviewEvidenceCanonicalDir(dir string) string { } // reviewEvidenceDirs is dir followed by its .uid fallback siblings. +// +// The siblings are found by listing dir's parent (the repo directory), which +// in a hosted hive holds one entry per PR ever reviewed — thousands, on NFS. +// A Glob per lookup made every tracker build, which looks up one bundle per +// open PR, re-read that directory once per PR (hivecommons/hive#14681). The +// listing is memoised per parent for evidenceSiblingListTTL and dropped when +// this process creates a sibling itself; a sibling another writer creates +// becomes visible within the TTL. func reviewEvidenceDirs(dir string) []string { dirs := []string{dir} - matches, _ := filepath.Glob(dir + evidenceFallbackDirMarker + "*") - sort.Strings(matches) - for _, m := range matches { - if _, err := strconv.Atoi(strings.TrimPrefix(m, dir+evidenceFallbackDirMarker)); err != nil { + base := filepath.Base(dir) + for _, name := range evidenceSiblingNames(filepath.Dir(dir)) { + rest, ok := strings.CutPrefix(name, base+evidenceFallbackDirMarker) + if !ok { + continue + } + if _, err := strconv.Atoi(rest); err != nil { continue } + m := filepath.Join(filepath.Dir(dir), name) if info, err := os.Stat(m); err == nil && info.IsDir() { dirs = append(dirs, m) } @@ -609,6 +624,62 @@ func reviewEvidenceDirs(dir string) []string { return dirs } +// evidenceSiblingListTTL bounds how long a repo directory's list of +// .uid fallback siblings is reused before it is read again. +const evidenceSiblingListTTL = 60 * time.Second + +// evidenceSiblingListMaxParents caps the memo; past it the whole memo is +// dropped rather than evicted entry by entry (a hive has tens of repos). +const evidenceSiblingListMaxParents = 512 + +type evidenceSiblingList struct { + names []string // entries containing evidenceFallbackDirMarker, sorted + at time.Time +} + +var ( + evidenceSiblingMu sync.Mutex + evidenceSiblingLists = map[string]evidenceSiblingList{} + // evidenceSiblingNow is the clock the memo ages by; tests replace it. + evidenceSiblingNow = time.Now +) + +// evidenceSiblingNames is the sorted names under parent that contain the +// fallback marker, from the memo when it is younger than the TTL. +func evidenceSiblingNames(parent string) []string { + now := evidenceSiblingNow() + evidenceSiblingMu.Lock() + if l, ok := evidenceSiblingLists[parent]; ok && now.Sub(l.at) < evidenceSiblingListTTL { + evidenceSiblingMu.Unlock() + return l.names + } + evidenceSiblingMu.Unlock() + var names []string + if entries, err := os.ReadDir(parent); err == nil { + for _, e := range entries { + if strings.Contains(e.Name(), evidenceFallbackDirMarker) { + names = append(names, e.Name()) + } + } + } + sort.Strings(names) + evidenceSiblingMu.Lock() + if len(evidenceSiblingLists) >= evidenceSiblingListMaxParents { + evidenceSiblingLists = map[string]evidenceSiblingList{} + } + evidenceSiblingLists[parent] = evidenceSiblingList{names: names, at: now} + evidenceSiblingMu.Unlock() + return names +} + +// forgetEvidenceSiblings drops the memoised sibling list for the parent of +// dir; called when this process creates a fallback sibling. +func forgetEvidenceSiblings(dir string) { + evidenceSiblingMu.Lock() + delete(evidenceSiblingLists, filepath.Dir(dir)) + evidenceSiblingMu.Unlock() +} + // resolveReviewEvidencePath is the most recently written copy of the bundle // at path across the PR directory and its fallback siblings, or path itself // when no copy exists (so callers still see os.ErrNotExist). diff --git a/src/pkg/github/review_evidence_siblings_test.go b/src/pkg/github/review_evidence_siblings_test.go new file mode 100644 index 0000000000..16ec9e4eaa --- /dev/null +++ b/src/pkg/github/review_evidence_siblings_test.go @@ -0,0 +1,70 @@ +package github + +import ( + "os" + "path/filepath" + "reflect" + "testing" + "time" +) + +// TestReviewEvidenceDirsMemoisesTheRepoListing pins the #14681 fix: one +// repo-directory read serves every PR lookup for the TTL, a sibling this +// process writes is visible at once, and one another writer creates shows +// up once the TTL lapses. +func TestReviewEvidenceDirsMemoisesTheRepoListing(t *testing.T) { + root := t.TempDir() + repo := filepath.Join(root, "o", "r") + for _, d := range []string{"7", "7.uid1001", "8", "8.uidx"} { + if err := os.MkdirAll(filepath.Join(repo, d), 0o755); err != nil { + t.Fatal(err) + } + } + clock := time.Date(2026, 10, 27, 20, 0, 0, 0, time.UTC) + evidenceSiblingNow = func() time.Time { return clock } + t.Cleanup(func() { + evidenceSiblingNow = time.Now + forgetEvidenceSiblings(filepath.Join(repo, "7")) + }) + + dir7, dir8 := filepath.Join(repo, "7"), filepath.Join(repo, "8") + if got := reviewEvidenceDirs(dir7); !reflect.DeepEqual(got, []string{dir7, filepath.Join(repo, "7.uid1001")}) { + t.Fatalf("dirs(7) = %v", got) + } + // A non-numeric suffix is not a sibling. + if got := reviewEvidenceDirs(dir8); !reflect.DeepEqual(got, []string{dir8}) { + t.Fatalf("dirs(8) = %v", got) + } + + // Another writer adds a sibling: invisible until the TTL lapses. + if err := os.MkdirAll(filepath.Join(repo, "8.uid2002"), 0o755); err != nil { + t.Fatal(err) + } + if got := reviewEvidenceDirs(dir8); len(got) != 1 { + t.Fatalf("dirs(8) within TTL = %v, want the memoised listing", got) + } + clock = clock.Add(evidenceSiblingListTTL) + if got := reviewEvidenceDirs(dir8); !reflect.DeepEqual(got, []string{dir8, filepath.Join(repo, "8.uid2002")}) { + t.Fatalf("dirs(8) after TTL = %v", got) + } + + // This process creating one drops the memo for its parent. + if err := os.MkdirAll(filepath.Join(repo, "7.uid3003"), 0o755); err != nil { + t.Fatal(err) + } + forgetEvidenceSiblings(dir7) + if got := reviewEvidenceDirs(dir7); len(got) != 3 { + t.Fatalf("dirs(7) after forget = %v, want both siblings", got) + } + + // The memo is bounded: past the cap it is dropped wholesale, not grown. + for i := 0; i < evidenceSiblingListMaxParents+1; i++ { + evidenceSiblingNames(filepath.Join(root, "missing", string(rune('a'+i%26)), time.Duration(i).String())) + } + evidenceSiblingMu.Lock() + n := len(evidenceSiblingLists) + evidenceSiblingMu.Unlock() + if n > evidenceSiblingListMaxParents { + t.Fatalf("memo holds %d parents, cap %d", n, evidenceSiblingListMaxParents) + } +}