Skip to content

hive pod RSS grows ~360 MiB/min (hive Go process, not agent CLIs) and OOM-kills at 24Gi (twice on 2026-10-27) #14681

Closed

Description

@clubanderson

Summary

The hive Go process (PID 1 child hive --config /etc/hive/hive.yaml) in the hosted hive hive-hosted-hosted-available-oke-11-placeholder-r05x (ctx hive-oke) leaks memory at roughly 360 MiB/min on the current candidate image (commit ee64433, #14617). The pod OOM-killed twice today (exit 137 at the 24Gi limit: ~14:55Z and 17:15:52Z). The owner raised the limit to 40Gi at 18:58Z; at the observed rate the new pod reaches that at roughly 20:40Z.

The growth is in the hive binary itself, not in the agent CLIs. Per-process RSS sampled in-pod (/proc/<pid>/status) on the fresh pod hive-b96dbc8c5-7cpdn (started 19:00Z):

UTC hive RSS 14 copilot CLIs + wrappers (sum) hive threads pod (kubectl top)
19:03 0.49 GiB 3.5 GiB 22 4.6 GiB
19:07 1.12 GiB 5.5 GiB 25 6.4 GiB
19:11 2.48 GiB 5.9 GiB 47 7.9 GiB
19:15 4.33 GiB 5.8 GiB 61 9.7 GiB
19:19 5.61 GiB 6.0 GiB 118 11.4 GiB
19:23 8.00 GiB 5.6 GiB 128 12.9 GiB
19:28 9.10 GiB 5.6 GiB 156 14.9 GiB
  • Agent CLI processes are flat: each copilot-linux-arm64/copilot sits at 310–540 MiB (median 350 MiB); pluk watch ≈ 57 MiB each; node server.js 92 MiB. They are a constant ~6 GiB baseline, not the growth.
  • Page cache (total_cache) is ~0.9 GiB and flat. RssAnon of the hive process = 6.3 GiB at 19:17 (RssFile 63 MiB) → it is anonymous Go heap, not mmap'd files.
  • RSS at 2 s resolution oscillates ±200–450 MiB (GC cycles) around a rising baseline — so the live heap is growing, i.e. retention, not just GC slack. Growth rate accelerates with uptime (≈80 MiB/min in min 0–2, ≈250 MiB/min in min 5–10, ≈360 MiB/min after min 15), which smells like an O(n)-per-tick walk/re-marshal over an append-only in-memory collection.
  • Leak rate also got worse across today's candidate builds: pod hive-5bfc9d6589-pl4pr ran 02:50Z→14:55Z before its first OOM (~28 MiB/min), the restarted container ran 14:55Z→17:15Z (~100 MiB/min), the current build does ~360 MiB/min. Suspect: something merged between 2026-10-27 02:50Z and 18:15Z (see the commit window below).
  • CPU of the hive process is steady at ~4.8 cores (3.1 user + 1.7 sys, from /proc/1447/stat deltas). The sys share plus the monotonically growing OS thread count (22 → 156 in 25 min; Go never releases Ms) means many goroutines are concurrently parked in blocking syscalls — regular-file I/O (note /data is NFS 10.0.10.30:/…) or os/exec — not sockets (/proc/net/tcp* shows only ~100 sockets total, flat).
  • Activity during the window was ordinary: copilot usage recorded 94/5 min, agent output signal 91/5 min, automerge skip 76/5 min, webhook re-read patched the cached actionable snapshot 27/5 min, review-request watcher: verdict recorded 15/5 min, 14 agents launched, 5 kicks. No webhook/connection pile-up.

Previous pod's logs were not retrievable (--previous pod already gone when checked).

What I ruled out (read origin/v5 src)

  • webhook re-read patched the cached actionable snapshot — cmd/hive/eval_cycle_source.go:95-120 is copy-on-write (Clone() per patch, old snapshot becomes garbage). Churn, not retention.
  • Agent manager per-agent slices: KickHistory capped (manager_kick.go:459), RestartEvents pruned (manager_restart_telemetry.go:27,59), lastPaneCapture replaced each 3 s poll (manager_poll.go:106), pane poll is a bounded capture-pane -S -2000 per agent every 3 s.
  • Copilot sniff proxy request/response buffers (crux/proxy/github_proxy.go:3551, :3657, 32 MiB cap each) are per-request and dropped after relay — unless a goroutine per connection is stuck holding them (the copilot sniff: upstream response read timed out warnings fit, but socket count is flat).
  • Evidence bundles are written to disk (pkg/github/review_evidence.go:551), not retained.
  • Agent CLI processes themselves (flat, see table).

Ranked suspects (unconfirmed — no pprof available)

  1. Something in the hive binary that holds a growing in-memory collection and re-walks or re-serialises it on a timer, consistent with the accelerating rate and ~1.7 cores of sys time. Highest-rate timer work during the window: the self-authored automerge sweep (pkg/github/automerge/automerge_sweep.go:1161, every ~60 s across all repos → automerge skip 76/5 min), the stage-label reconciler (stage labels: reviewer verdict route 22/5 min), the review-request watcher / verdict store (review-verdicts.json is 7.3 MiB and rewritten minutes apart), and the dashboard status builder (owner has the dashboard open; pkg/dashboard/status_builder.go is polled continuously, and ✨ Live per-run cost · tokens · elapsed · tests · files line on agent cards #14580 / 🐛 fix(dashboard): show the fleet running · $/hour figure in the visible Governor grid #14597 added per-run cost + fleet $/hour computation to it today). Candidate commits in the window that touch retained state: ✨ review: delta re-review mode for PRs already judged at an earlier head #14622 (delta re-review), 🐛 fix(review): expire stale pending fixes and route decision requests before conflicts #14624 (pending-fix expiry), 🐛 fix(review): give held PRs a body digest so head verdicts converge #14601 (held-PR body digest in verdict store), ✨ Live per-run cost · tokens · elapsed · tests · files line on agent cards #14580/🐛 fix(dashboard): show the fleet running · $/hour figure in the visible Governor grid #14597 (dashboard per-run cost aggregation), ✨ House-style section every kick opens with: repo HIVE_STYLE.md > hive default > built-in #14582 (house-style section read every kick), ✨ feat(automerge): approve held workflow runs for registry-trusted contributors #14520/✨ feat(automerge): contributor auto-promotion counts clean merges across the hive's repos #14518 (held workflow run approvals / contributor promotion counting across repos).
  2. Blocking-syscall goroutine pile-up (thread count 22→156): goroutines stuck in file I/O on the NFS-backed /data (JSON stores rewritten per event: review-verdicts.json, pr-claims.json, stage-labels.json, review-outcomes.json, launch-cost.json, prompt-history.jsonl 11 MiB append-only) or in os/exec waits, each pinning its request buffers / snapshots. If each stuck goroutine holds a multi-MiB marshalled copy, 100+ of them is a few hundred MiB — a contributor, probably not the whole 9 GiB.
  3. Copilot-sniff proxy connections whose relay goroutine never exits after upstream response read timed out (github_proxy.go:3575-3590), each holding a ≤32 MiB request body + bufio readers. Socket count being flat argues against it being primary, but the bodies are large (agents send full context each turn, ~1 MiB+) and 94 completions/5 min would make this ~100 MiB/min if even half leaked.

Proposed fix / next steps

  1. Observability first (blocks any real fix): there is no net/http/pprof, expvar, or runtime.ReadMemStats anywhere in src/ (verified by grep; docs/general-technical-review.md:360 says the same). Add net/http/pprof on a loopback-only admin listener (or behind the dashboard owner auth) plus a periodic runtime.ReadMemStats/NumGoroutine INFO log line (every 60 s: HeapInuse, HeapAlloc, NumGC, goroutines, threads). With that, two heap + goroutine profiles 10 min apart on this pod will name the leak in one shot. Also log GOMEMLIMIT.
  2. Mitigation now: set GOMEMLIMIT (e.g. 16GiB) on the container so the GC runs harder before the cgroup limit, and consider a GOGC lower than 100. This will not fix a true retention leak but buys time and turns an OOM into a slow-down; if RSS still climbs linearly under GOMEMLIMIT it is a hard proof of retention rather than GC slack.
  3. Bisect the build window: redeploy the candidate from before 2026-10-27 02:50Z (the ~28 MiB/min build) vs the current one and compare the 15-minute hive-process RSS slope; then bisect the ~60 commits between. The rate tripled twice in one day, which should make the culprit commit stand out.
  4. Audit the timer-driven stores listed under suspect 1 for unbounded append + full re-marshal per tick, and bound/evict them (per-head TTL, size cap).

Repro / measurement commands used

kubectl --context hive-oke -n hive-hosted-hosted-available-oke-11-placeholder-r05x top pod -l app=hive   # every 60 s
kubectl ... exec <pod> -- sh -c 'awk "/VmRSS|Threads/" /proc/<hive-pid>/status'                             # every 2 s / 2 min
kubectl ... exec <pod> -- sh -c 'for p in /proc/[0-9]*; do ...VmRSS per cmdline...; done'                  # per-process breakdown
kubectl ... exec <pod> -- cat /sys/fs/cgroup/memory/memory.stat                                            # total_rss vs total_cache

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.
    on Oct 27, 2026
  2. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 27, 2026
  3. github-actions commented on Oct 27, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions