Skip to content

convergence: mutation journal grows without bound (53k entries / 56 MB, rewritten on every transition) #14827

Closed

Description

@clubanderson

Summary

pkg/convergence/mutation.Journal keeps every terminal (Applied / NotApplied) operation forever. On the hosted hive /data/convergence/mutation/journal.json is now 53,336 entries / 56 MB (growing ~4k entries/day), and every Begin, RecordResult, Reconcile and Get:

  1. re-reads and json.Unmarshals the whole 56 MB file (reloadLocked),
  2. clone()s all 53k operations into a fresh map,
  3. clones them all again and MarshalIndents the full snapshot back to disk (persistLocked).

Evidence (pprof on the hosted pod, image with #14682)

go tool pprof -top -cum -sample_index=alloc_space allocs.pb.gz (23 min uptime, 525 GB allocated ≈ 22 GB/min):

314 GB 59.8%  mutation.(*Boundary).Execute
172 GB 32.8%  mutation.(*Journal).persistLocked   (via commit)
115 GB 21.9%  mutation.(*Journal).reloadLocked    (via lockAndRefreshLocked)
 17 GB  3.4%  mutation.Operation.clone

inuse_space carries a persistent ~44 MB Operation.clone residue (the in-memory index) plus the transient 56 MB marshal buffer per transition. GC runs ~60 cycles/min and total_sys sits at 1.86 GiB against ~300 MiB live heap — this churn is the main reason the Go process holds ~2× its live heap.

Proposed fix

Bound the journal at commit time:

  • terminal entries whose UpdatedAt is older than DefaultJournalRetention (7 d) are dropped;
  • past DefaultJournalMaxEntries (20k) the oldest terminal entries are evicted first;
  • Planned / Unknown entries are never pruned — they still need reconciliation.

Logical IDs embed DesiredGeneration and ClaimKey, so a week-old terminal entry cannot be replayed by a live planner; the idempotency guarantee for in-flight work is unchanged.

Follow-up (separate): Ledger (claims.json, 7 MB) has the same keep-forever shape for released claims.

Refs #14681.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 28, 2026
  2. github-actions commented on Oct 28, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghas-open-prhelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions