Repository navigation
π fix(github): routine since= reads get free 304s within the hour, and the hive names the sweep behind each REST call - #17101
Conversation
β¦ sweeps name their REST calls Several sweeps built their list query's since from time.Now() on every pass (un-park comment reads, closed-issue stage sweep and review backlog split, shared-CI rerun, same-component sibling check). The URL changed on every call, so the ETag cache, keyed on the full URL, never matched and every pass was charged. The since sent to GitHub is now rounded down to the UTC hour (cacheableSince); callers keep filtering on their exact cutoff, so the wider window misses nothing and answers nothing twice. Untagged REST calls from the sweep workers, the governor cycle, the forgestate loop, the metrics collector, the request watchers, the mention poller and dashboard requests now carry WithRESTCaller names, so /api/gh-rate-limits no longer lumps ~4,400 calls an hour under "hive". Refs hivecommons#17075 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Read the full diff against the v6 tree at head 37dda40 and every caller of the two rounded list helpers. Looks correct to me.
What I verified
cacheableSince(src/pkg/github/etagcache.go:193) only ever movessinceearlier, by < 1h, and keeps zero as zero.etagCacheKey(etagcache.go:173) keys onreq.URL.String(), so a stablesinceis exactly what makes the repeat a304.- Every
listIssueCommentsSincecaller re-filters on its exact cutoff, so the wider window cannot answer an old command:attention_command.go:47βcm.GetCreatedAt().Time.Before(cutoff)skippr_approve_command.go:25β same skipissue_park_recheck.go:672βd < -parkCopyNoteWindowskippr_level_hold.go:324passes zerosince, unaffected
- Both
listClosedIssuesSincecallers (closed_issue_stage_sweep.go:83,review_backlog_split.go:237) do not re-filter, but each is idempotent on label/ledger state β an already-cleaned issue hitslen(stage) == 0 && !needsHuman && len(retired) == 0 β continue, so re-reading β€1h more closes costs a loop iteration, not an action. WithRESTCaller(restnegativecache.go:31) is plaincontext.WithValue, so the innermost tag wins: the outerhive:eval_cycleinrunEvalCycledoes not maskhive:enumerate_actionableetc., andsweepJob.runTaggeddoes not mask sweeps that tag more narrowly.
Body and diff agree. One note, not a defect: the body says since is now cache-stable "within the hour", which also means a sweep's first pass after each top-of-hour boundary is a full charge across all 20 repos at once β expected, and still strictly better than today.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from Read the tree at The "callers keep their exact cutoff" claim holds. I opened every call site of the two helpers that now send a rounded
Caller tagging. Checks I ran on the head: One pre-existing gap, not introduced here: |
Summary
Several sweeps send GitHub a
since=built from the current clock on every pass, such as "comments from the last 7 days". The URL changes every time, so the ETag cache, which is keyed on the full URL, never matches and every pass is charged. This PR rounds thesincesent to GitHub down to the hour, so every pass in that hour asks for the same URL and gets a free304when nothing changed. Each sweep still filters on its exact cutoff, so nothing is missed. The PR also tags the untagged callers, so/api/gh-rate-limitsnames the sweep behind each call instead of lumping about 4,400 calls an hour under plainhive.Refs #17075
What went wrong
The knuckle hive (20 repos) needs about 8,250 GitHub calls an hour against a 7,000 limit, and it runs out most hours. In
ghRateLimits.top_consumersfor 18:54:40β19:45:33Z on 2026-11-07 (about 50 minutes), the top rows were:hiveGET /issues/{number}/commentshiveGET /issueshiveGET /issues/{number}hiveGET /issues/{number}/timelinehiveGET /pullshiveGET /pulls/{number}/reviewsA
304from the ETag cache costs nothing. The pull-request lists are mostly free. The issue reads are mostly charged.On the v6 tip, these list URLs carry a
sincethat moves on every pass:SweepIssueUnparkCommands, every 30 s to 5 min). The 7-day cutoff istime.Now()-based. It feedslistIssueCommentsSincefor/hive approvereplies on every open PR with recent comments, and for/hive ack/snoozeon items with an open attention episode.time.Now() - 48h, and later passes uselastRun - 10m. That moves on every pass too.now - 72h.now - 14d.go-github writes
sinceto the second, so any two passes more than a second apart missed the cache.I checked the cache side too.
etagCacheKeyis the identity hash, plusAccept, plusreq.URL.String(), so the query is part of the key. All of these lists go through the App client'sgithubTransportChain, which includes the ETag cache. No hot path here bypasses it.What changes
cacheableSince(inetagcache.go) rounds a list query'ssincedown to the UTC hour. A zerosincestays zero. The rounding happens where the list URL is built:listIssueCommentsSince,listClosedIssuesSince, the shared-CI incident list and the same-component sibling list. Each caller keeps its exact cutoff for its own filtering.Why a wider window is harmless. The window only gets wider, by less than an hour. The
/hive approveand attention-command paths still drop comments created before their exact cutoff. The extra comments they now see can only add reply markers to the dedupe set, so no old command is answered again. The closed-issue sweep and the backlog split re-read a few closes they already handled: the first finds no stage labels left, and the second is deduped by its ledger keys. The shared-CI and sibling lists filterclosed_aton the exact time.IssueUnparkSweepOptions.Nowis a test clock, the same pattern asSharedCIRerunOptions.Now.Caller tags (
WithRESTCaller):hive:<sweep name>, for examplehive:issue_unpark,hive:closed_issue_stage,hive:task_listorhive:supersession;hive:eval_cycle, withhive:enumerate_actionable,hive:duplicate_pr_guard,hive:ledger_cover,hive:escalation_sweep,hive:hold_guard,hive:pr_follow_ups,hive:hard_suppress_reconcileandhive:park_recheckfor its larger steps;hive:forgestateand the dashboard metricshive:metrics_collector;hive:issue_request_watcherandhive:pr_request_watcher, and the mention pollerhive:mention_poll;hive:dashboard_api.A call already tagged more narrowly keeps its own name. No call site was refactored.
Expected saving (estimate)
This is an estimate of about 700β1,500 charged calls an hour on the knuckle hive. Most of it should come from the
/hive approvecomment reads, which hit every open PR on every un-park pass. The rest comes from the closed-issue, backlog-split and shared-CI lists. After this change, each URL should be charged about once an hour, plus whenever the data really changes.How to check after deploy: compare the
not_modifiedshare ofGET /issues/{number}/commentsandGET /issuesinghRateLimits.top_consumers(fleet_statusor/api/gh-rate-limits) with the table above. The rows will now be split by caller, so add up the rows for each endpoint.hive:issue_unparkandhive:closed_issue_stageshould be mostly304. Plainhiveshould be a small remainder.Files
src/pkg/github/etagcache.go:cacheableSinceand why it exists.src/pkg/github/pr_level_hold.go,closed_issue_stage_sweep.go,shared_ci_rerun.go,issue_request_watcher.go: round thesincethat is sent.src/pkg/github/issue_unpark_command.go: theNowtest clock.src/cmd/hive/sweep_workers.go,main_eval_cycle.go,forgestate_boot.go,src/pkg/dashboard/server.go,metrics_collector.go,src/pkg/github/{issue,pr}_request_watcher.go,mention_comments.go: caller tags.src/docs/operator-reference/sweeps-and-operations.md: a note next to the API-budget text.src/docs/design/agent-turn-model.md: twomain_eval_cycle.goline cites, which were already stale on tip, now point at the right lines.changelog.d/fixed-cacheable-since-reads.md.Tests
TestCacheableSince: zero stays zero, rounding goes down to the UTC hour (including from a +5:30 zone), and the result is never later than the input or more than an hour earlier.TestSweepIssueUnparkCommands_CommentListSinceIsCacheable: three un-park passes against a fake server with ETag/304handling. A pass 40 minutes later in the same hour requests the identical comments URL,sinceincluded, and gets a304. A pass in the next hour movessinceforward to that hour.TestSweepClosedIssueStageLabels_ListSinceIsCacheable: the same check for the closed-issue list, using the scheduler's movinglastRun - 10m.TestSweepJobRunTaggedAttributesRESTCalls(cmd/hive): a sweep job's untagged timeline read shows up in REST accounting ashive:<job>.TestSweepPRApproveRepliesnow expects the roundedsince.On v6 tip, all four new tests fail. The un-park test was adapted to the real clock with a 1.1 s gap, and the tag test calls
job.runthe way the worker did. The second pass gotsince=β¦:54and200instead of the first pass's URL and304, and the timeline read was recorded as callerhive. With this change they pass. I ran the full./pkg/github/...,./pkg/dashboard/,./pkg/writeback/and./cmd/hive/suites,go vetand the docs link and citation checks.π€ Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1