Repository navigation
π convergence: bound mutation journal so memory stops growing - #14828
Conversation
The operation journal (journal.json) kept every Applied/NotApplied entry forever. On the hosted hive it reached 53k entries / 56 MB and, because every Begin/RecordResult/Reconcile reloads the file, clones every entry and rewrites the whole snapshot with MarshalIndent, it accounted for ~60% of all Go allocations (~13 GB/min) and a persistent ~44 MB heap residue. Terminal entries now age out after DefaultJournalRetention (7 days) and the oldest terminal entries are evicted past DefaultJournalMaxEntries (20k). Planned/Unknown entries are never pruned: they still need reconciliation. Logical IDs embed the desired generation and claim key, so a week-old terminal entry can no longer be replayed by a live planner. Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
There was a problem hiding this comment.
Style/maintainability review (read at head 4d12ffb) β no blockers.
One advisory: commit's doc comment (src/pkg/convergence/mutation/journal.go:523-524, "rolling back the in-memory index when the durable write fails") is now only partially true β the rollback at :531-535 restores the committed entry but not the entries pruneLocked deleted at :529. That's harmless (the next transition's lockAndRefreshLocked reloads from disk, so memory is never authority), but a future reader of the rollback path will wonder. Suggest saying so:
// commit installs the updated operation, prunes the index, and persists
// atomically, rolling back the committed entry when the durable write fails.
// Pruned entries are not restored: the next transition reloads the on-disk
// snapshot, which still holds them. Callers hold j.mu.
β hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Intent alignment (read at 4d12ffb): the diff does what #14827 asks β retention + max-entries pruning in commit under both locks, Planned/Unknown exempt, no scope creep, no Ledger changes. One in-scope gap between the stated safety argument and the code:
The "cannot be replayed" premise does not hold for Boundary effects β the ones filling the journal. The PR body and the new doc comment (src/pkg/convergence/mutation/journal.go:97-101) justify 7-day pruning because "every logical ID embeds the outcome's desired generation and the authorizing claim key, so a terminal entry this old can no longer be replayed". But boundaryEffect hardcodes DesiredGeneration: 1 and derives ClaimKey deterministically from TaskClaim(repo, subject) (src/pkg/convergence/mutation/boundary.go:272-290), so a Boundary logical ID is a pure function of (repo, kind, target, inputs). For e.g. CreateIssueComment the inputs are just the body digest (src/pkg/github/client.go:2731-2735); review submits and status creates likewise carry no time bucket (client.go:2895-2897, 2990-2992). Today an identical comment/review re-requested on the same PR is refused forever via ErrAlreadyApplied β idempotent success (boundary.go:113-120). After this PR, once the Applied entry is >7 days old the same request runs the effect again: ExecuteReconciling only observes external state on ErrNeedsReconciliation, a missing entry goes straight to effect (src/crux/effects/effects.go:126-130). So any sweep that re-issues an identical comment/review/status on a long-lived PR more than a week apart will now produce a visible duplicate. The one caller that already reasons about this (issue_request_watcher.go:1535-1544) explicitly windows itself at 24h because "the journal otherwise remembers an applied create forever" β other producers were written against that forever guarantee.
Requested change: don't ship the misleading invariant. Either (a) state the real contract and accept one possible duplicate per identical effect per retention window, confirming no periodic path re-issues identical comment/review bodies >7d apart on the same target, or (b) keep maxEntries as the only bound (zero retention) so the oldest-terminal eviction still caps the file without a time-based cliff. For (a), at minimum:
// DefaultJournalRetention bounds how long a TERMINAL operation (Applied or
// NotApplied) stays in the journal after its last transition. Boundary
// logical IDs are a pure function of (repo, kind, target, inputs) β see
// boundaryEffect β so pruning trades the forever idempotency guarantee for
// a bounded file: an IDENTICAL effect re-requested more than retention after
// it last applied runs again. Callers that must suppress a repeat for longer
// must observe external state first or widen their idempotency window.
// Planned and Unknown entries are never pruned: they still need
// reconciliation against authoritative external state.
(replace journal.go:97-104, and the same sentence in the PR body / changelog fragment).
β hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Security review (read at 4d12ffb)
1. (medium, in-scope) The "cannot be replayed" rationale is false for every Boundary-mediated effect; pruning reopens the exact-ID replay fence.
journal.go:97-105 and the PR body justify dropping Applied entries because logical IDs "embed the outcome's desired generation and the authorizing claim key". But boundaryEffect (boundary.go:277-293) hardcodes DesiredGeneration: 1 and sets ClaimKey: TaskClaim(repo, repo#kind/target).Key(), which claim.go:103-111 renders as a static prefix+subject. The ID for a label/comment/review/merge/PR-create is therefore a pure function of (repo, kind, target, claim.Inputs) and is identical a week later. Once pruned, Boundary.Execute β Journal.Begin mints a fresh Planned op and runs the effect; ExecuteReconciling (src/crux/effects/effects.go:126-139) only observes external state on ErrNeedsReconciliation, not on a missing entry, and LookupApplied (boundary.go:263-266) returns false. Only issue-create carries a time bucket (issue_request_watcher.go:1592); PR-create (pullrequest.go:313) and the others do not. Net effect: the durable idempotency guarantee becomes a sliding window. That may be an acceptable trade per #14827, but the code comment and PR body should state it as a bounded fence rather than claim replay is impossible.
2. (medium, in-scope) Count eviction has no age floor, so a burst can strip the fence from in-flight work β contradicting "idempotency for in-flight work is unchanged".
pruneLocked (journal.go:561-582) evicts the oldest terminal entries by UpdatedAt with no minimum age. At the stated ~4k entries/day, 20k is reached in ~5 days (already inside the 7-day retention), and a burst of transitions on a public repo (externally triggered label/comment churn) can evict Applied PR-create/comment records that are minutes old while a retry loop still references them (boundary.go:112-124 relies on ErrAlreadyApplied for idempotent success). Suggest never count-evicting entries younger than the unresolved TTL:
@@ func (j *Journal) pruneLocked(now time.Time) {
- terminal := make([]*Operation, 0, len(j.ops))
- for _, op := range j.ops {
- if isTerminal(op.Status) {
+ // Never count-evict a terminal entry still young enough for an in-flight
+ // retry to depend on its ErrAlreadyApplied fence.
+ floor := now.Add(-journalEvictionMinAge)
+ terminal := make([]*Operation, 0, len(j.ops))
+ for _, op := range j.ops {
+ if isTerminal(op.Status) && op.UpdatedAt.Before(floor) {
terminal = append(terminal, op)
}
}with e.g. const journalEvictionMinAge = 24 * time.Hour next to DefaultJournalMaxEntries, and reword journal.go:97-105 to describe the bounded window.
Advisory (out-of-scope): pruneLocked(work.UpdatedAt) trusts the caller-supplied clock (journal.go:529); a far-future now from any caller would purge all terminal entries. All current callers pass executor.now(), so no change needed now.
β hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. |
There was a problem hiding this comment.
Correctness review (read at head 4d12ffb).
1. The retention rationale does not hold for the production caller β idempotency becomes timeβbounded (medium, inβscope).
journal.go:97-101 (and the PR body) justify pruning Applied entries with "every logical ID embeds the outcome's desired generation and the authorizing claim key, so a terminal entry this old can no longer be replayed by a live planner." But the only production Effect constructor, boundaryEffect (boundary.go:282-290), hardcodes DesiredGeneration: 1 and derives ClaimKey from TaskClaim(repo, subject).Key() (claim.go:135, claim.go:103) β both constant for a given repo/target. So the real logical ID is hash(repo, kind, target, inputs), and a replay with identical inputs after 7 days will reach Begin as a new entry instead of ErrAlreadyApplied (journal.go:386-389) / LookupApplied (boundary.go:263-266). For labels/merge this is externally idempotent; for KindIssueComment the ID is just the body digest (client.go:2731-2735) and #14011 explicitly relied on "every later post of the same comment was refused forever" β after this PR an identical comment/review body re-planned on a longβlived PR >7 days later is posted again. That may be an acceptable trade (the issue proposes it), but the docstring must state the real guarantee rather than a false one so nobody builds on it:
// DefaultJournalRetention bounds how long a TERMINAL operation (Applied or
// NotApplied) stays in the journal after its last transition. NOTE: the
// production Boundary derives logical IDs from (repo, kind, target, inputs)
// only (boundaryEffect fixes DesiredGeneration=1 and a per-subject claim
// key), so pruning time-bounds the "same effect never runs twice" guarantee
// to this window: an identical effect re-planned after retention is treated
// as new. Externally idempotent kinds (labels, merge, PR-create via
// open-PR-by-head lookup) are unaffected; comment/review bodies may re-post.
// Planned and Unknown entries are never pruned: they still need
// reconciliation against authoritative external state.
2. Eviction can drop the entry being committed (low, inβscope).
pruneLocked (journal.go:553-583) is called from commit after j.ops[id] = &cp (journal.go:528-529) and ranks terminal entries by UpdatedAt. If β₯ maxEntries entries carry a later UpdatedAt than the caller's now (clock skew between processes sharing the file, or a caller passing a stale instant), the just-committed terminal op is evicted, yet commit returns it as success and the next Get misses. Cheap guard:
- j.pruneLocked(work.UpdatedAt)
+ j.pruneLocked(id, work.UpdatedAt)
...
-func (j *Journal) pruneLocked(now time.Time) {
+func (j *Journal) pruneLocked(keep string, now time.Time) {
if j.retention > 0 {
cutoff := now.Add(-j.retention)
for id, op := range j.ops {
- if isTerminal(op.Status) && op.UpdatedAt.Before(cutoff) {
+ if id != keep && isTerminal(op.Status) && op.UpdatedAt.Before(cutoff) {
delete(j.ops, id)
}
}
}
...
for _, op := range j.ops {
- if isTerminal(op.Status) {
+ if op.LogicalID != keep && isTerminal(op.Status) {
terminal = append(terminal, op)
}
}Checked and fine: rollback on persist failure (journal.go:530-535) leaves pruned entries gone from memory only, but every entry point reloads from disk first (reloadLocked, journal.go:260-288), so disk stays authority. Tests use fixed instants and cover retention, eviction order, Planned/Unknown survival and zero bounds.
β hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 4d12ffb. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
There was a problem hiding this comment.
Read journal.go at 4d12ffb plus the Begin callers (mutation/boundary.go, publish/publisher.go, audit/audit.go) and one automerge consumer. The pruning itself is sound: pruneLocked runs under j.mu + the flock inside commit (journal.go:529), only touches Applied/NotApplied (journal.go:545), the just-committed entry carries UpdatedAt = now so it is never the eviction victim, and a failed persistLocked is harmless because every transition reloads from disk first. Looks correct to me as a bound.
One premise in the body and in the new doc comment does not hold on the main production path (journal.go:97-102: "every logical ID embeds the outcome's desired generation and the authorizing claim key, so a terminal entry this old can no longer be replayed by a live planner"):
mutation/boundary.go:284βboundaryEffecthardcodesDesiredGeneration: 1, and itsClaimKeyisTaskClaim(repo, repo#kind/target).Key()(boundary.go:273,303,claim.go:110) β a pure function of the claim, no time component.publish/publisher.go:77βpublicationGeneration = 1likewise.
So for every effects.Execute call (all of pkg/github/automerge/*, cruxhost/host.go:139), the logical ID is sha256(repo, 1, external.<kind>, repo#kind/target, claimKey, kind, inputs) and is identical forever for the same inputs β e.g. automerge_held_runs.go:405 keys a comment on {head_sha, body digest} only. A PR whose head sits unchanged for more than 7 days has its Applied record pruned, after which Begin (journal.go:382) accepts the same effect as new and boundary.go:263 (Replayed) returns false.
In practice this is defence-in-depth, not the sole guard: the one consumer I traced pre-checks GitHub for its own marker (automerge_held_runs.go:223 β :322) before calling Execute. I did not trace the other ~10 effects.Claim{} sites, so I am stating this as an open question rather than a defect: is every boundary caller expected to re-observe external state before re-issuing an effect? If yes, the comment at journal.go:97-102 and the body should say that (the bound is safe because callers pre-check, not because IDs rotate); if any caller relies on the journal alone, it will re-fire after 7 days of a stable head.
Minor: reloadLocked (journal.go:276-284) still verifies every entry's ID and rejects duplicates, so a snapshot pruned by one process is accepted cleanly by the others β no change needed there.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Fixes #14827
Refs #14681
What
Bound
pkg/convergence/mutation.Journal, the durable idempotency record behind every label / comment / review / merge / PR-create effect:Applied,NotApplied) age out afterDefaultJournalRetention(7 days);DefaultJournalMaxEntries(20 000) the oldest terminal entries are evicted first;Planned/Unknownentries are never pruned β they still need reconciliation against external state.Pruning happens in
commitbeforepersistLocked, under the journal mutex and cross-process flock, so the next on-disk snapshot (which every other process reloads) reflects the bound. Zero values disable each bound.Why
On the hosted hive
journal.jsonis 53 336 entries / 56 MB and grows ~4k entries/day. Every transition reloads + clones the whole index and rewrites it withMarshalIndent, so the journal was ~60 % of all Go allocations (~13 GB/min) in the pprofallocsprofile taken with #14682, plus a persistent ~44 MBOperation.cloneheap residue. Logical IDs embedDesiredGenerationandClaimKey, so a week-old terminal entry can no longer be replayed by a live planner; idempotency for in-flight work is unchanged.Tests
journal_prune_test.go(notime.Sleep; fixed instants):TestJournal_PrunesTerminalEntriesPastRetentionβ stale Applied dropped (memory + persisted snapshot), Planned and fresh Applied kept;TestJournal_EvictsOldestTerminalEntriesPastMaxEntriesβ assertslen(ops) <= maxEntriesafter every commit, oldest terminal evicted first, Unknown survives;TestJournal_ZeroBoundsDisablePruning.go build ./...,gofmt -lclean,go test ./pkg/convergence/mutation/β 90.9 % coverage.