Skip to content

πŸ› fix(github): routine since= reads get free 304s within the hour, and the hive names the sweep behind each REST call - #17101

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:fix/cacheable-since-reads
Nov 7, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:fix/cacheable-since-reads

Conversation

@Danathar

@Danathar Danathar commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Several sweeps send GitHub a since= built from the current clock on every pass, such as "comments from the last 7 days". The URL changes every time, so the ETag cache, which is keyed on the full URL, never matches and every pass is charged. This PR rounds the since sent to GitHub down to the hour, so every pass in that hour asks for the same URL and gets a free 304 when nothing changed. Each sweep still filters on its exact cutoff, so nothing is missed. The PR also tags the untagged callers, so /api/gh-rate-limits names the sweep behind each call instead of lumping about 4,400 calls an hour under plain hive.

Refs #17075

What went wrong

The knuckle hive (20 repos) needs about 8,250 GitHub calls an hour against a 7,000 limit, and it runs out most hours. In ghRateLimits.top_consumers for 18:54:40–19:45:33Z on 2026-11-07 (about 50 minutes), the top rows were:

Caller Endpoint Requests Charged Answered 304
hive GET /issues/{number}/comments 2776 1357 51%
hive GET /issues 1399 1012 28%
hive GET /issues/{number} 823 677 18%
hive GET /issues/{number}/timeline 910 675 26%
hive GET /pulls 2048 343 83%
hive GET /pulls/{number}/reviews 2910 261 91%

A 304 from the ETag cache costs nothing. The pull-request lists are mostly free. The issue reads are mostly charged.

On the v6 tip, these list URLs carry a since that moves on every pass:

  • Un-park sweep (SweepIssueUnparkCommands, every 30 s to 5 min). The 7-day cutoff is time.Now()-based. It feeds listIssueCommentsSince for /hive approve replies on every open PR with recent comments, and for /hive ack/snooze on items with an open attention episode.
  • Closed-issue stage sweep and review backlog split (every 10 min Γ— 20 repos). The first pass uses time.Now() - 48h, and later passes use lastRun - 10m. That moves on every pass too.
  • Shared-CI rerun sweep (every 15 min). Incidents closed since now - 72h.
  • Same-component check when an agent files an issue. Bot issues closed since now - 14d.

go-github writes since to the second, so any two passes more than a second apart missed the cache.

I checked the cache side too. etagCacheKey is the identity hash, plus Accept, plus req.URL.String(), so the query is part of the key. All of these lists go through the App client's githubTransportChain, which includes the ETag cache. No hot path here bypasses it.

What changes

  • cacheableSince (in etagcache.go) rounds a list query's since down to the UTC hour. A zero since stays zero. The rounding happens where the list URL is built: listIssueCommentsSince, listClosedIssuesSince, the shared-CI incident list and the same-component sibling list. Each caller keeps its exact cutoff for its own filtering.

  • Why a wider window is harmless. The window only gets wider, by less than an hour. The /hive approve and attention-command paths still drop comments created before their exact cutoff. The extra comments they now see can only add reply markers to the dedupe set, so no old command is answered again. The closed-issue sweep and the backlog split re-read a few closes they already handled: the first finds no stage labels left, and the second is deduped by its ledger keys. The shared-CI and sibling lists filter closed_at on the exact time.

  • IssueUnparkSweepOptions.Now is a test clock, the same pattern as SharedCIRerunOptions.Now.

  • Caller tags (WithRESTCaller):

    • every periodic sweep worker becomes hive:<sweep name>, for example hive:issue_unpark, hive:closed_issue_stage, hive:task_list or hive:supersession;
    • the governor cycle becomes hive:eval_cycle, with hive:enumerate_actionable, hive:duplicate_pr_guard, hive:ledger_cover, hive:escalation_sweep, hive:hold_guard, hive:pr_follow_ups, hive:hard_suppress_reconcile and hive:park_recheck for its larger steps;
    • the state cache loop becomes hive:forgestate and the dashboard metrics hive:metrics_collector;
    • the issue and PR request watchers become hive:issue_request_watcher and hive:pr_request_watcher, and the mention poller hive:mention_poll;
    • calls made while serving a dashboard request become hive:dashboard_api.

    A call already tagged more narrowly keeps its own name. No call site was refactored.

Expected saving (estimate)

This is an estimate of about 700–1,500 charged calls an hour on the knuckle hive. Most of it should come from the /hive approve comment reads, which hit every open PR on every un-park pass. The rest comes from the closed-issue, backlog-split and shared-CI lists. After this change, each URL should be charged about once an hour, plus whenever the data really changes.

How to check after deploy: compare the not_modified share of GET /issues/{number}/comments and GET /issues in ghRateLimits.top_consumers (fleet_status or /api/gh-rate-limits) with the table above. The rows will now be split by caller, so add up the rows for each endpoint. hive:issue_unpark and hive:closed_issue_stage should be mostly 304. Plain hive should be a small remainder.

Files

  • src/pkg/github/etagcache.go: cacheableSince and why it exists.
  • src/pkg/github/pr_level_hold.go, closed_issue_stage_sweep.go, shared_ci_rerun.go, issue_request_watcher.go: round the since that is sent.
  • src/pkg/github/issue_unpark_command.go: the Now test clock.
  • src/cmd/hive/sweep_workers.go, main_eval_cycle.go, forgestate_boot.go, src/pkg/dashboard/server.go, metrics_collector.go, src/pkg/github/{issue,pr}_request_watcher.go, mention_comments.go: caller tags.
  • src/docs/operator-reference/sweeps-and-operations.md: a note next to the API-budget text. src/docs/design/agent-turn-model.md: two main_eval_cycle.go line cites, which were already stale on tip, now point at the right lines.
  • changelog.d/fixed-cacheable-since-reads.md.

Tests

  • TestCacheableSince: zero stays zero, rounding goes down to the UTC hour (including from a +5:30 zone), and the result is never later than the input or more than an hour earlier.
  • TestSweepIssueUnparkCommands_CommentListSinceIsCacheable: three un-park passes against a fake server with ETag/304 handling. A pass 40 minutes later in the same hour requests the identical comments URL, since included, and gets a 304. A pass in the next hour moves since forward to that hour.
  • TestSweepClosedIssueStageLabels_ListSinceIsCacheable: the same check for the closed-issue list, using the scheduler's moving lastRun - 10m.
  • TestSweepJobRunTaggedAttributesRESTCalls (cmd/hive): a sweep job's untagged timeline read shows up in REST accounting as hive:<job>.
  • TestSweepPRApproveReplies now expects the rounded since.

On v6 tip, all four new tests fail. The un-park test was adapted to the real clock with a 1.1 s gap, and the tag test calls job.run the way the worker did. The second pass got since=…:54 and 200 instead of the first pass's URL and 304, and the timeline read was recorded as caller hive. With this change they pass. I ran the full ./pkg/github/..., ./pkg/dashboard/, ./pkg/writeback/ and ./cmd/hive/ suites, go vet and the docs link and citation checks.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

… sweeps name their REST calls

Several sweeps built their list query's since from time.Now() on every
pass (un-park comment reads, closed-issue stage sweep and review backlog
split, shared-CI rerun, same-component sibling check). The URL changed on
every call, so the ETag cache, keyed on the full URL, never matched and
every pass was charged. The since sent to GitHub is now rounded down to
the UTC hour (cacheableSince); callers keep filtering on their exact
cutoff, so the wider window misses nothing and answers nothing twice.

Untagged REST calls from the sweep workers, the governor cycle, the
forgestate loop, the metrics collector, the request watchers, the mention
poller and dashboard requests now carry WithRESTCaller names, so
/api/gh-rate-limits no longer lumps ~4,400 calls an hour under "hive".

Refs hivecommons#17075

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 7, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 7, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff against the v6 tree at head 37dda40 and every caller of the two rounded list helpers. Looks correct to me.

What I verified

  • cacheableSince (src/pkg/github/etagcache.go:193) only ever moves since earlier, by < 1h, and keeps zero as zero. etagCacheKey (etagcache.go:173) keys on req.URL.String(), so a stable since is exactly what makes the repeat a 304.
  • Every listIssueCommentsSince caller re-filters on its exact cutoff, so the wider window cannot answer an old command:
    • attention_command.go:47 β€” cm.GetCreatedAt().Time.Before(cutoff) skip
    • pr_approve_command.go:25 β€” same skip
    • issue_park_recheck.go:672 β€” d < -parkCopyNoteWindow skip
    • pr_level_hold.go:324 passes zero since, unaffected
  • Both listClosedIssuesSince callers (closed_issue_stage_sweep.go:83, review_backlog_split.go:237) do not re-filter, but each is idempotent on label/ledger state β€” an already-cleaned issue hits len(stage) == 0 && !needsHuman && len(retired) == 0 β†’ continue, so re-reading ≀1h more closes costs a loop iteration, not an action.
  • WithRESTCaller (restnegativecache.go:31) is plain context.WithValue, so the innermost tag wins: the outer hive:eval_cycle in runEvalCycle does not mask hive:enumerate_actionable etc., and sweepJob.runTagged does not mask sweeps that tag more narrowly.

Body and diff agree. One note, not a defect: the body says since is now cache-stable "within the hour", which also means a sweep's first pass after each top-of-hour boundary is a full charge across all 20 repos at once β€” expected, and still strictly better than today.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the tree at 37dda40, not just the diff. This looks correct to me.

The "callers keep their exact cutoff" claim holds. I opened every call site of the two helpers that now send a rounded since:

  • listIssueCommentsSince β†’ attention_command.go:47 drops CreatedAt.Before(cutoff); pr_approve_command.go:25 drops CreatedAt.Before(cutoff); issue_park_recheck.go:673-676 bounds on closedAt Β± parkCopyNoteWindow; pr_level_hold.go:324 passes a zero since, which cacheableSince (etagcache.go:193) leaves zero.
  • listClosedIssuesSince β†’ closed_issue_stage_sweep.go:83 and review_backlog_split.go:237 do not re-filter on time, but both are idempotent over already-handled closes (label clear / host split keyed on a fixed set), so the <1h widening only re-reads work already done.
  • issue_request_watcher.go:2301 and shared_ci_rerun.go:199 keep their closed_at filter on the exact cutoff.

Caller tagging. WithRESTCaller is a plain context.WithValue (restnegativecache.go:31), so a narrower inner tag shadows the outer hive:eval_cycle / hive:<sweep> one as the comments say. sweep_workers.go:48 wraps every job uniformly.

Checks I ran on the head: go build ./pkg/github/ ./pkg/dashboard/ ./cmd/hive/ clean; check-api-reference-citations.sh and check-docs-citations.py pass (2007 refs) β€” the two agent-turn-model.md line repoints are correct against the current main_eval_cycle.go:1094/1580, which were already stale at merge-base.

One pre-existing gap, not introduced here: pr_approve_command_test.go:43 now accepts any hour-aligned since in a one-hour band, so a regression that stopped rounding would still fail (good), but one that rounded to the day would also pass that check β€” TestCacheableSince in cacheable_since_test.go covers the granularity directly, so this is fine as-is.

Evidence bundle `#1710137dda40` sha256:5dd1b359361bfc7461a74b2f45aaca94794d93226774520e4963a692505aeb8b β€” download from the dashboard or `hivectl review evidence #17101`.

@hivecommons-hive
hivecommons-hive Bot merged commit dd6b372 into hivecommons:v6 Nov 7, 2026
69 of 70 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 37dda40.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant