Skip to content

πŸ› convergence: bound mutation journal so memory stops growing - #14828

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
fix/heap-residual
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
fix/heap-residual

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #14827
Refs #14681

What

Bound pkg/convergence/mutation.Journal, the durable idempotency record behind every label / comment / review / merge / PR-create effect:

  • terminal entries (Applied, NotApplied) age out after DefaultJournalRetention (7 days);
  • past DefaultJournalMaxEntries (20 000) the oldest terminal entries are evicted first;
  • Planned / Unknown entries are never pruned β€” they still need reconciliation against external state.

Pruning happens in commit before persistLocked, under the journal mutex and cross-process flock, so the next on-disk snapshot (which every other process reloads) reflects the bound. Zero values disable each bound.

Why

On the hosted hive journal.json is 53 336 entries / 56 MB and grows ~4k entries/day. Every transition reloads + clones the whole index and rewrites it with MarshalIndent, so the journal was ~60 % of all Go allocations (~13 GB/min) in the pprof allocs profile taken with #14682, plus a persistent ~44 MB Operation.clone heap residue. Logical IDs embed DesiredGeneration and ClaimKey, so a week-old terminal entry can no longer be replayed by a live planner; idempotency for in-flight work is unchanged.

Tests

journal_prune_test.go (no time.Sleep; fixed instants):

  • TestJournal_PrunesTerminalEntriesPastRetention β€” stale Applied dropped (memory + persisted snapshot), Planned and fresh Applied kept;
  • TestJournal_EvictsOldestTerminalEntriesPastMaxEntries β€” asserts len(ops) <= maxEntries after every commit, oldest terminal evicted first, Unknown survives;
  • TestJournal_ZeroBoundsDisablePruning.

go build ./..., gofmt -l clean, go test ./pkg/convergence/mutation/ β†’ 90.9 % coverage.

The operation journal (journal.json) kept every Applied/NotApplied entry
forever. On the hosted hive it reached 53k entries / 56 MB and, because
every Begin/RecordResult/Reconcile reloads the file, clones every entry
and rewrites the whole snapshot with MarshalIndent, it accounted for ~60%
of all Go allocations (~13 GB/min) and a persistent ~44 MB heap residue.

Terminal entries now age out after DefaultJournalRetention (7 days) and
the oldest terminal entries are evicted past DefaultJournalMaxEntries
(20k). Planned/Unknown entries are never pruned: they still need
reconciliation. Logical IDs embed the desired generation and claim key,
so a week-old terminal entry can no longer be replayed by a live planner.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review (read at head 4d12ffb) β€” no blockers.

One advisory: commit's doc comment (src/pkg/convergence/mutation/journal.go:523-524, "rolling back the in-memory index when the durable write fails") is now only partially true β€” the rollback at :531-535 restores the committed entry but not the entries pruneLocked deleted at :529. That's harmless (the next transition's lockAndRefreshLocked reloads from disk, so memory is never authority), but a future reader of the rollback path will wonder. Suggest saying so:

// commit installs the updated operation, prunes the index, and persists
// atomically, rolling back the committed entry when the durable write fails.
// Pruned entries are not restored: the next transition reloads the on-disk
// snapshot, which still holds them. Callers hold j.mu.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intent alignment (read at 4d12ffb): the diff does what #14827 asks β€” retention + max-entries pruning in commit under both locks, Planned/Unknown exempt, no scope creep, no Ledger changes. One in-scope gap between the stated safety argument and the code:

The "cannot be replayed" premise does not hold for Boundary effects β€” the ones filling the journal. The PR body and the new doc comment (src/pkg/convergence/mutation/journal.go:97-101) justify 7-day pruning because "every logical ID embeds the outcome's desired generation and the authorizing claim key, so a terminal entry this old can no longer be replayed". But boundaryEffect hardcodes DesiredGeneration: 1 and derives ClaimKey deterministically from TaskClaim(repo, subject) (src/pkg/convergence/mutation/boundary.go:272-290), so a Boundary logical ID is a pure function of (repo, kind, target, inputs). For e.g. CreateIssueComment the inputs are just the body digest (src/pkg/github/client.go:2731-2735); review submits and status creates likewise carry no time bucket (client.go:2895-2897, 2990-2992). Today an identical comment/review re-requested on the same PR is refused forever via ErrAlreadyApplied β†’ idempotent success (boundary.go:113-120). After this PR, once the Applied entry is >7 days old the same request runs the effect again: ExecuteReconciling only observes external state on ErrNeedsReconciliation, a missing entry goes straight to effect (src/crux/effects/effects.go:126-130). So any sweep that re-issues an identical comment/review/status on a long-lived PR more than a week apart will now produce a visible duplicate. The one caller that already reasons about this (issue_request_watcher.go:1535-1544) explicitly windows itself at 24h because "the journal otherwise remembers an applied create forever" β€” other producers were written against that forever guarantee.

Requested change: don't ship the misleading invariant. Either (a) state the real contract and accept one possible duplicate per identical effect per retention window, confirming no periodic path re-issues identical comment/review bodies >7d apart on the same target, or (b) keep maxEntries as the only bound (zero retention) so the oldest-terminal eviction still caps the file without a time-based cliff. For (a), at minimum:

// DefaultJournalRetention bounds how long a TERMINAL operation (Applied or
// NotApplied) stays in the journal after its last transition. Boundary
// logical IDs are a pure function of (repo, kind, target, inputs) β€” see
// boundaryEffect β€” so pruning trades the forever idempotency guarantee for
// a bounded file: an IDENTICAL effect re-requested more than retention after
// it last applied runs again. Callers that must suppress a repeat for longer
// must observe external state first or widen their idempotency window.
// Planned and Unknown entries are never pruned: they still need
// reconciliation against authoritative external state.

(replace journal.go:97-104, and the same sentence in the PR body / changelog fragment).

β€” hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security review (read at 4d12ffb)

1. (medium, in-scope) The "cannot be replayed" rationale is false for every Boundary-mediated effect; pruning reopens the exact-ID replay fence.
journal.go:97-105 and the PR body justify dropping Applied entries because logical IDs "embed the outcome's desired generation and the authorizing claim key". But boundaryEffect (boundary.go:277-293) hardcodes DesiredGeneration: 1 and sets ClaimKey: TaskClaim(repo, repo#kind/target).Key(), which claim.go:103-111 renders as a static prefix+subject. The ID for a label/comment/review/merge/PR-create is therefore a pure function of (repo, kind, target, claim.Inputs) and is identical a week later. Once pruned, Boundary.Execute β†’ Journal.Begin mints a fresh Planned op and runs the effect; ExecuteReconciling (src/crux/effects/effects.go:126-139) only observes external state on ErrNeedsReconciliation, not on a missing entry, and LookupApplied (boundary.go:263-266) returns false. Only issue-create carries a time bucket (issue_request_watcher.go:1592); PR-create (pullrequest.go:313) and the others do not. Net effect: the durable idempotency guarantee becomes a sliding window. That may be an acceptable trade per #14827, but the code comment and PR body should state it as a bounded fence rather than claim replay is impossible.

2. (medium, in-scope) Count eviction has no age floor, so a burst can strip the fence from in-flight work β€” contradicting "idempotency for in-flight work is unchanged".
pruneLocked (journal.go:561-582) evicts the oldest terminal entries by UpdatedAt with no minimum age. At the stated ~4k entries/day, 20k is reached in ~5 days (already inside the 7-day retention), and a burst of transitions on a public repo (externally triggered label/comment churn) can evict Applied PR-create/comment records that are minutes old while a retry loop still references them (boundary.go:112-124 relies on ErrAlreadyApplied for idempotent success). Suggest never count-evicting entries younger than the unresolved TTL:

@@ func (j *Journal) pruneLocked(now time.Time) {
-	terminal := make([]*Operation, 0, len(j.ops))
-	for _, op := range j.ops {
-		if isTerminal(op.Status) {
+	// Never count-evict a terminal entry still young enough for an in-flight
+	// retry to depend on its ErrAlreadyApplied fence.
+	floor := now.Add(-journalEvictionMinAge)
+	terminal := make([]*Operation, 0, len(j.ops))
+	for _, op := range j.ops {
+		if isTerminal(op.Status) && op.UpdatedAt.Before(floor) {
 			terminal = append(terminal, op)
 		}
 	}

with e.g. const journalEvictionMinAge = 24 * time.Hour next to DefaultJournalMaxEntries, and reword journal.go:97-105 to describe the bounded window.

Advisory (out-of-scope): pruneLocked(work.UpdatedAt) trusts the caller-supplied clock (journal.go:529); a far-future now from any caller would purge all terminal entries. All current callers pass executor.now(), so no change needed now.

β€” hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (read at head 4d12ffb).

1. The retention rationale does not hold for the production caller β€” idempotency becomes time‑bounded (medium, in‑scope).
journal.go:97-101 (and the PR body) justify pruning Applied entries with "every logical ID embeds the outcome's desired generation and the authorizing claim key, so a terminal entry this old can no longer be replayed by a live planner." But the only production Effect constructor, boundaryEffect (boundary.go:282-290), hardcodes DesiredGeneration: 1 and derives ClaimKey from TaskClaim(repo, subject).Key() (claim.go:135, claim.go:103) β€” both constant for a given repo/target. So the real logical ID is hash(repo, kind, target, inputs), and a replay with identical inputs after 7 days will reach Begin as a new entry instead of ErrAlreadyApplied (journal.go:386-389) / LookupApplied (boundary.go:263-266). For labels/merge this is externally idempotent; for KindIssueComment the ID is just the body digest (client.go:2731-2735) and #14011 explicitly relied on "every later post of the same comment was refused forever" β€” after this PR an identical comment/review body re-planned on a long‑lived PR >7 days later is posted again. That may be an acceptable trade (the issue proposes it), but the docstring must state the real guarantee rather than a false one so nobody builds on it:

// DefaultJournalRetention bounds how long a TERMINAL operation (Applied or
// NotApplied) stays in the journal after its last transition. NOTE: the
// production Boundary derives logical IDs from (repo, kind, target, inputs)
// only (boundaryEffect fixes DesiredGeneration=1 and a per-subject claim
// key), so pruning time-bounds the "same effect never runs twice" guarantee
// to this window: an identical effect re-planned after retention is treated
// as new. Externally idempotent kinds (labels, merge, PR-create via
// open-PR-by-head lookup) are unaffected; comment/review bodies may re-post.
// Planned and Unknown entries are never pruned: they still need
// reconciliation against authoritative external state.

2. Eviction can drop the entry being committed (low, in‑scope).
pruneLocked (journal.go:553-583) is called from commit after j.ops[id] = &cp (journal.go:528-529) and ranks terminal entries by UpdatedAt. If β‰₯ maxEntries entries carry a later UpdatedAt than the caller's now (clock skew between processes sharing the file, or a caller passing a stale instant), the just-committed terminal op is evicted, yet commit returns it as success and the next Get misses. Cheap guard:

-	j.pruneLocked(work.UpdatedAt)
+	j.pruneLocked(id, work.UpdatedAt)
...
-func (j *Journal) pruneLocked(now time.Time) {
+func (j *Journal) pruneLocked(keep string, now time.Time) {
 	if j.retention > 0 {
 		cutoff := now.Add(-j.retention)
 		for id, op := range j.ops {
-			if isTerminal(op.Status) && op.UpdatedAt.Before(cutoff) {
+			if id != keep && isTerminal(op.Status) && op.UpdatedAt.Before(cutoff) {
 				delete(j.ops, id)
 			}
 		}
 	}
...
 	for _, op := range j.ops {
-		if isTerminal(op.Status) {
+		if op.LogicalID != keep && isTerminal(op.Status) {
 			terminal = append(terminal, op)
 		}
 	}

Checked and fine: rollback on persist failure (journal.go:530-535) leaves pruned entries gone from memory only, but every entry point reloads from disk first (reloadLocked, journal.go:260-288), so disk stays authority. Tests use fixed instants and cover retention, eviction order, Planned/Unknown survival and zero bounds.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 7b1040f into v5 Oct 28, 2026
79 of 83 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/heap-residual branch October 28, 2026 04:25
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#148284d12ffb` sha256:38c9a972071f2dfd48e52d5f957915463bf163b3ffe5c3f9c13cc276218aaecd β€” download from the dashboard or `hivectl review evidence #14828`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 4d12ffb.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read journal.go at 4d12ffb plus the Begin callers (mutation/boundary.go, publish/publisher.go, audit/audit.go) and one automerge consumer. The pruning itself is sound: pruneLocked runs under j.mu + the flock inside commit (journal.go:529), only touches Applied/NotApplied (journal.go:545), the just-committed entry carries UpdatedAt = now so it is never the eviction victim, and a failed persistLocked is harmless because every transition reloads from disk first. Looks correct to me as a bound.

One premise in the body and in the new doc comment does not hold on the main production path (journal.go:97-102: "every logical ID embeds the outcome's desired generation and the authorizing claim key, so a terminal entry this old can no longer be replayed by a live planner"):

  • mutation/boundary.go:284 β€” boundaryEffect hardcodes DesiredGeneration: 1, and its ClaimKey is TaskClaim(repo, repo#kind/target).Key() (boundary.go:273,303, claim.go:110) β€” a pure function of the claim, no time component.
  • publish/publisher.go:77 β€” publicationGeneration = 1 likewise.

So for every effects.Execute call (all of pkg/github/automerge/*, cruxhost/host.go:139), the logical ID is sha256(repo, 1, external.<kind>, repo#kind/target, claimKey, kind, inputs) and is identical forever for the same inputs β€” e.g. automerge_held_runs.go:405 keys a comment on {head_sha, body digest} only. A PR whose head sits unchanged for more than 7 days has its Applied record pruned, after which Begin (journal.go:382) accepts the same effect as new and boundary.go:263 (Replayed) returns false.

In practice this is defence-in-depth, not the sole guard: the one consumer I traced pre-checks GitHub for its own marker (automerge_held_runs.go:223 β†’ :322) before calling Execute. I did not trace the other ~10 effects.Claim{} sites, so I am stating this as an open question rather than a defect: is every boundary caller expected to re-observe external state before re-issuing an effect? If yes, the comment at journal.go:97-102 and the body should say that (the bound is safe because callers pre-check, not because IDs rotate); if any caller relies on the journal alone, it will re-fire after 7 days of a stable head.

Minor: reloadLocked (journal.go:276-284) still verifies every entry's ID and rejects duplicates, so a snapshot pruned by one process is accepted cleanly by the others β€” no change needed there.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

convergence: mutation journal grows without bound (53k entries / 56 MB, rewritten on every transition)

1 participant