Repository navigation
π fix(automerge): cache contributor merged-repo search across sweeps - #15371
Conversation
#15125 memoized contributorMergedRepos within one sweep and skipped bot authors. A human author outside the registry was still searched again on every sweep, 3-4 search/issues calls per author on a 19-repo hive, even after they had passed auto-promotion. The per-author answer (which active repositories hold a merge by them) is now kept on the Engine for an hour, keyed by the active-repository set. A failed search is kept for 15 minutes, so a secondary rate limit is not re-tripped every sweep; the PR's own repository is counted alone meanwhile, as before. The per-sweep context cache is subsumed and removed. Only the list is cached. The PR's own repository and every listed one are still walked fresh each sweep for clean merges and revert/sentinel evidence, so a stale list can only leave out a repository the author first merged into within the hour. Refs #15088 Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
There was a problem hiding this comment.
Correctness β one in-scope finding (reviewed head 748175c).
The cache does not survive sweeps on the governor run-loop path
The cache is a field on Engine (src/pkg/github/automerge/automerge_sweep.go:336-337), but the contributor lane runs from two places in production, and one of them builds a throwaway Engine every sweep:
cmd/hive/run_loop_deps.go:89βrunAutoMergeSweepIfDueβautomerge.SweepContributorLaneAutoMerges(ctx, ghClient, opts, β¦)atcmd/hive/main.go:10052βreturn New(transport, opts).SweepContributorLaneAutoMerges(ctx, sweepOpts)atsrc/pkg/github/automerge/automerge_sweep.go:492-493.
On that path c.contributorMergedReposCache is nil at the start of every sweep, so a human author outside the registry is still searched on every governor sweep β the exact behaviour the PR title and summary say is fixed ("This PR keeps the result across sweeps"). Only the long-lived self-authored loop (sweepPass, automerge_sweep.go:1684) benefits. The failure TTL is likewise lost there, so a 403 is re-tripped per sweep as before.
The package already documents this engine-per-call shape and the fix pattern: automerge_stage_labels.go:218-220 ("One tracker is shared process-wide (sharedStageTracker) because the governor sweep builds a fresh engine per lane") with stageTrackerOrShared() at :338-343. The tests can't catch this because every test uses a single Engine instance.
Suggested shape (tests keep an isolated cache via the Engine field; production falls through to the shared one):
--- a/src/pkg/github/automerge/automerge_contributor_lane.go
+++ b/src/pkg/github/automerge/automerge_contributor_lane.go
@@
type contributorMergedReposEntry struct {
refs []contributorRepoRef
repos string
expires time.Time
}
+
+// contributorMergedReposCache is shared process-wide by default because the
+// governor sweep builds a fresh Engine per lane (automerge_sweep.go:492,
+// cmd/hive/main.go:10052); see sharedStageTracker for the same reason.
+type contributorMergedReposCache struct {
+ mu sync.Mutex
+ entries map[string]contributorMergedReposEntry
+}
+
+var sharedContributorMergedReposCache = &contributorMergedReposCache{}
+
+func (c *Engine) contributorMergedReposCacheOrShared() *contributorMergedReposCache {
+ if c != nil && c.contributorMergedRepos != nil {
+ return c.contributorMergedRepos
+ }
+ return sharedContributorMergedReposCache
+}
@@ func (c *Engine) contributorMergedRepos(
- c.contributorMergedReposMu.Lock()
- entry, ok := c.contributorMergedReposCache[key]
- c.contributorMergedReposMu.Unlock()
+ cache := c.contributorMergedReposCacheOrShared()
+ cache.mu.Lock()
+ entry, ok := cache.entries[key]
+ cache.mu.Unlock()
if ok && entry.repos == repos && now.Before(entry.expires) {
return others(entry.refs)
}
remember := func(refs []contributorRepoRef, ttl time.Duration) {
- c.contributorMergedReposMu.Lock()
- defer c.contributorMergedReposMu.Unlock()
- if c.contributorMergedReposCache == nil || len(c.contributorMergedReposCache) >= contributorMergedReposCacheMax {
- c.contributorMergedReposCache = map[string]contributorMergedReposEntry{}
+ cache.mu.Lock()
+ defer cache.mu.Unlock()
+ if cache.entries == nil || len(cache.entries) >= contributorMergedReposCacheMax {
+ cache.entries = map[string]contributorMergedReposEntry{}
}
- c.contributorMergedReposCache[key] = contributorMergedReposEntry{refs: refs, repos: repos, expires: now.Add(ttl)}
+ cache.entries[key] = contributorMergedReposEntry{refs: refs, repos: repos, expires: now.Add(ttl)}
}
--- a/src/pkg/github/automerge/automerge_sweep.go
+++ b/src/pkg/github/automerge/automerge_sweep.go
@@ type Engine struct {
- contributorMergedReposMu sync.Mutex
- contributorMergedReposCache map[string]contributorMergedReposEntry
+ // contributorMergedRepos overrides the process-shared cache (tests).
+ contributorMergedRepos *contributorMergedReposCacheThen have newContributorLaneEngineRepos (test helper) set contributorMergedRepos: &contributorMergedReposCache{} so tests stay isolated, and add one test that calls the package-level SweepContributorLaneAutoMerges (or New(...) twice) and asserts fx.searches == 1 across the two engines β that is the case the current suite cannot see.
Minor, same area: with a shared cache the now := c.clock() used for expires must be comparable across engines; both paths use real time in production, so that holds, but worth a comment.
β hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Reviewed (docs-currency) β no blocking findings. Verified src/docs/scale-envelope.md:137 now cites automerge_sweep.go:505, which holds DefaultAutoMergeSweepMaxMerges at head 748175c.
One advisory, non-blocking: changelog.d/fixed-15088-promotion-search-ttl.md:1 states the TTL but not the operator-visible trade-off documented in the code comment (automerge_contributor_lane.go on contributorMergedReposTTL): a repository the author first merged into during the last hour is counted, and its reverts seen, up to an hour late. changelog.d/README.md asks fragments to say "why an operator cares", so consider:
- Contributor auto-promotion now remembers each author's merged-repository search for an hour (a failed search for 15 minutes) instead of searching again on every sweep, so a promoted human author no longer trips secondary rate limits sweep after sweep ([#15088](https://github.com/hivecommons/hive/issues/15088)). Every listed repository is still walked fresh each sweep; only a repository the author first merged into within the last hour can be counted up to an hour late.
β hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@Danathar this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Summary
Contributor auto-promotion searched for an author's merges on every sweep. On a 19-repo hive that is 3 to 4
search/issuescalls per author, per sweep.Evidence from hosted-projectbluefin-knuckle-gjvq, running f950551 (before #15125):
contributorMergedReposranis:pr is:merged author:<login> repo:β¦ repo:β¦for every contributor-lane PR on every sweep.github-actions[bot](utah-packages#763) this caused 16 secondary 403s in 13 hours.#15125 fixed the bot case. It skips
[bot]and App logins, and it reuses one author's result within a sweep. A human author outside the registry was still searched again on the next sweep. That includes authors who already passed auto-promotion. This PR keeps the result across sweeps.Refs #15088
Change
automerge_contributor_lane.go,automerge_sweep.go). The search result for each author is kept in a mutex-guarded map onEnginefor 1 hour. The result is the list of active repositories that hold a merge by the author.Engine.clock()(the injectablenow), so the tests step time forward.scale-envelope.mdcitedautomerge_sweep.go:375forDefaultAutoMergeSweepMaxMerges. It is at line 505 after this change, and the citation now says so.changelog.d/fixed-15088-promotion-search-ttl.md.Why caching the list is safe
Only one thing is cached: which other repositories hold a merge by the author. I checked this in
contributorCleanMergedHistory:contributorCleanMergesInRepo. That walk counts clean merges and looks for reverts and the sentinel label.TestContributorPromotionSearchCacheWalksListedReposFreshchecks this.What a stale list can miss: a repository the author first merged into during the last hour. Its merges count up to an hour late. A revert of one of those merges is also seen up to an hour late. The walk already stopped at the first
needclean merges before this change, so it never read every repository.Not included, and why
admitContributorchecks the history again on each sweep. If the PR's own repository already has enough clean merges, no search runs at all. Otherwise the cache cuts it to one search per hour. A memo that skipped the whole check would also skip the fresh revert walk. A memo that kept the walk would save nothing beyond that one hourly search.Testing
From
src/,TZ=UTC,-count=1:go test ./pkg/github/automerge/ -run 'Contributor'passes, and so does the whole./pkg/github/automerge/package.go vet ./pkg/github/automerge/is clean, andgofmt -lis clean on the touched files.check-changelog-fragment.sh v6andcheck-docs-citations.pypass.Tests:
TestContributorPromotionSearchCachePerSweep(from π fix: limit contributor promotion searches per sweepΒ #15125, extended): 3 lookups for alice and 1 for bob make 2 searches. A later sweep 59 minutes on makes none. At 60 minutes alice is searched again. Its second half used to expect a new search on every sweep. That is the behaviour this PR changes, so the assertion moved.TestContributorPromotionSearchFailureCached: a 403 is remembered for 15 minutes, then searched again. The success after it is kept for the full hour.TestContributorPromotionSearchCacheIsPerAuthor: one search serves PRs inwidgetandhive, and each gets the other repository. A changed repository set searches again.TestContributorPromotionSearchCacheWalksListedReposFresh: a revert added after the search blocks promotion on the next sweep, with no new search.TestTrySweepContributorLanePRAutoPromotionScope: the query check changed from "must not include the PR's own repository" to "must include it", because the entry is per author.TestContributorPromotionSkipsBotsfrom π fix: limit contributor promotion searches per sweepΒ #15125 passes unchanged.Mutation checks: 8 mutants, each restored from a backup copy. Every one fails at least one test:
Still open
automerge_sweep.gocitations were already stale on v6. Inlabels-and-control-signals.mdanddesign/github-mention-triggers.md, the cited lines (987-1057, 1242) do not hold the code they describe. The citation check still passes, so I left them for a separate docs pass.Contributor checklist
changelog.d/fixed-15088-promotion-search-ttl.md.v6.src/docs/scale-envelope.md).π€ Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1