Skip to content

πŸ› fix(github): the un-park sweep reuses comment and timeline reads of items whose updated_at has not moved - #17174

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:fix/issue-unpark-skip-unchanged
Nov 8, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:fix/issue-unpark-skip-unchanged

Conversation

@Danathar

@Danathar Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

The issue un-park sweep re-read the comments (and, on label-approval repos, the label timeline) of every item it watches on every pass, about every five minutes, even though almost none of those items change between passes. It is the hive's largest REST consumer on the knuckle hive. The sweep now keeps the last comment list and label timeline for each item, keyed by the item's updated_at from the issue list it already reads, and reuses them until that timestamp moves. Every check that depends on the clock or live state still runs on every pass. Expected saving on knuckle is about 1,400–1,600 charged calls an hour.

What went wrong

ghRateLimits.top_consumers on hosted-projectbluefin-knuckle-gjvq (build dd6b372, read at 01:45:33Z on 2026-11-08; the window is the rate-limit hour that started 00:55:20Z):

caller endpoint requests charged 304
hive:issue_unpark GET issues/{n}/comments 1222 878 344
hive:issue_unpark GET issues/{n}/timeline 480 363 117

These came from 8 sweep passes (log: pass starts 00:55:37 to 01:38:10Z, about 73 s each). From 01:44Z the budget was critical and the sweep was skipped. That makes 1,241 charged calls in about 44 minutes of sweeping, around 1,700 an hour, or ~155 per pass.

Per pass, the sweep reads:

  • the full comment list of every parked issue, and of every issue that carries an approval label on a label-approval repo (tryUnparkIssue);
  • the label timeline of each approval-labelled issue (evaluateLabelApproval);
  • the since-list of every PR updated in the last week (replyPRApproveCommand);
  • the full list of every PR at 085/070 (tryUnparkPullRequest).

On knuckle that is about 219 items. I rebuilt every comment and timeline event on those 219 items for 00:00–02:00Z. Only 5 of them changed at all during the measured window. So nearly every charged read fetched data the sweep had already seen. The ETag cache did not catch those reads, for two reasons:

  • its key includes the token, and the installation token is refreshed about every 40 minutes;
  • it held only 504 entries, far under its 4,096-entry limit, so its 64 MiB byte limit is likely forcing evictions between passes.

What changes

  • New issue_unpark_read_cache.go keeps a per-item cache keyed by (repo, number) that holds updated_at, the comment list and the label events. The sweep reuses an entry only when all of these hold:
    • the listed updated_at is exactly the one the entry was read under;
    • the read happened at least 2 minutes after that updated_at, so a list and a read that briefly disagree right after a change can't pin a stale result;
    • the entry is less than 6 hours old (a backstop);
    • the sweep has not written to the item since.
  • The cache holds only raw data, never decisions. The seven-day command window, the commenter's live repo permission and the approval-label applier's trust are re-evaluated on every pass against the reused data, using the current clock.
  • An item is dropped from the cache when the sweep writes to it (reply, notice create/edit/delete, label change), when its processing errors, or when it leaves the repo's open list. The cache is capped at 4,096 items and 50,000 comments. It lives in memory only, so a restart starts empty, which is the old behaviour.
  • The four read sites go through the cache: tryUnparkIssue, tryUnparkPullRequest, replyPRApproveCommand, and the label-approval timeline read (evaluateLabelApprovalFrom, so other callers of evaluateLabelApproval are unchanged). Requests keep the hive:issue_unpark caller tag.

Why updated_at is enough. GitHub moves an issue's or PR's updated_at on every change these reads can see: a new or edited comment, and a label added or removed. I checked comment edits on projectbluefin. On common#1648, #1638 and #1644 the issue's updated_at equals the newest comment edit to the second. The only timeline events the sweep uses are labeled and unlabeled. One case I could not confirm without writing is whether deleting a comment moves updated_at. The 6-hour backstop covers it.

Expected saving

Steady state is the cache already warm when a new hour starts.

  • Changed items: 8 changed (item, pass) pairs across passes 2–8 of the window, about 11 comment reads, so roughly 15–20 reads an hour.
  • Backstop: about 219 items re-read every 6 h, so about 37 comment and 10 timeline reads an hour.
  • Total: about 60–70 reads an hour, against 1,483 an hour charged over the 50-minute clock window (1,700 an hour over the 44 minutes the sweep ran). That saves about 1,400–1,600 an hour.

A restart still costs one full pass (~215 reads), as it does today.

How to verify after deploy

In fleet_status β†’ .data.ghRateLimits.top_consumers, the hive:issue_unpark rows for GET /repos/{owner}/{repo}/issues/{number}/comments and .../timeline should drop from ~880 and ~360 charged per hour to a few dozen. The first pass after the restart that deploys this still reads everything once.

Files

  • src/pkg/github/issue_unpark_read_cache.go: the cache and the read helpers (new)
  • src/pkg/github/issue_unpark_command.go: scope, prune, and forget on write or error
  • src/pkg/github/pr_unpark_decision.go, pr_approve_command.go, label_approval.go: the read sites
  • src/pkg/github/client.go: the unparkReads field
  • changelog.d/fixed-issue-unpark-skip-unchanged.md

Tests

New in issue_unpark_read_cache_test.go. The first four fail on v6 tip 9975c5d and pass with this change:

  • TestUnparkSweepReusesReadsWhileUpdatedAtUnchanged: across two passes over a label-approval issue, a PR at 085 and a plain PR, the second pass makes no comment or timeline read.
  • TestUnparkSweepRereadsWhenUpdatedAtMoves: items whose updated_at moved are re-read; the unchanged one is not.
  • TestUnparkSweepTTLBackstopRereads: a read is reused at +5 h and re-read at +7 h.
  • TestUnparkSweepDecidesFromCachedCommentsWithCurrentState:
    • a commenter granted write access after pass 1 is answered on pass 2 from cached comments;
    • a command that slid outside the seven-day window on the current clock is not answered.
  • TestUnparkSweepRereadsJustUpdatedItem: the 2-minute settle rule. This is a guard, so it passes on tip as well.

issue_unpark_read_cache_internal_test.go covers pruning and both size bounds. I also updated TestSweepIssueUnparkCommands_CommentListSinceIsCacheable from #17101: it now drops the per-item cache between passes, because an unchanged item is no longer read at all and that test is about the URL a read sends.

I ran go test ./pkg/github/ ./pkg/writeback/ in full, go vet ./pkg/github/ and check-docs-citations.py, and all pass.

Refs #17101

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

…nchanged items

The issue un-park sweep re-read the comment list of every parked issue,
every PR waiting at 085/070 and every recently updated PR, plus the label
timeline of every approval-labelled issue, on every pass. Almost none of
those items change between passes, yet most reads were charged: on the
knuckle hive `hive:issue_unpark` made 1222 comment reads (878 charged) and
480 timeline reads (363 charged) in eight passes, the largest consumer.

The sweep now keeps the last comment list and label timeline per item,
keyed by the item's updated_at from the issue list it starts from, and
reuses them while updated_at has not moved. Only raw data is cached: the
seven-day command window, repo permission and label-trust checks still run
every pass. An item is read again when updated_at moves, when the read was
within two minutes of its last update, after the sweep writes to it, and at
least every six hours. The cache is in memory, pruned to listed items and
bounded by item and comment count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17169 πŸ› fix(automerge): a refused held-run approval is tried once per PR head, not again on every sweep same files src/pkg/github/client.go
#17130 ✨ feat(github): the hive files grouped issues from GitHub code scanning alerts (opt-in per repo) same files src/pkg/github/client.go

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read this against the v6 tree (issue_unpark_command.go, pr_approve_command.go, pr_unpark_decision.go, label_approval.go, pr_level_hold.go). The cache is sound as far as I can verify; one open question below that I could not settle from the code.

What holds up

  • Wider-than-asked cache hits are safe. unparkReadCache.comments returns a full list for any since, and an earlier-since list for a later one. The only since caller, replyPRApproveCommand, filters each comment on cm.GetCreatedAt().Time.Before(cutoff) itself (pr_approve_command.go:25), and the full-list callers (tryUnparkIssue, tryUnparkPullRequest) pass time.Time{}. So no caller sees comments it would not have filtered anyway. storeComments also refuses to overwrite a full list with a narrower one read later in the same pass, which is the right precedence.
  • Nothing decided is cached. evaluateLabelApprovalFrom is a pure seam: the existing evaluateLabelApproval delegates to it with the live listIssueLabelEvents, and the trust/permission checks still run on every pass over the events. The 7-day cutoff check stays in the caller.
  • Self-invalidation is complete for the sweep's own writes. Every write path (ensureUnparkNoticeComment create/move/edit/delete, tryUnparkIssue, tryUnparkPullRequest, replyPRApproveCommand, replyAttentionCommands) calls forgetUnparkRead on acted || err != nil, so a partial failure also forces a fresh read next pass rather than pinning a half-updated list.
  • Settle window and TTL do what the header comment says. usable rejects a read taken under 2 min after updated_at (so a list/comment race just after a change cannot pin stale data) and anything older than 6 h; readAt is the pass's now, which is earlier than the real read time, so the settle check errs conservative. prune only touches owner/repo# keys for the repo just listed; evictLocked is O(n) per store with n ≀ 4096, fine at ~220 items/pass.
  • Reads outside a sweep (no scope on ctx) go straight to GitHub, so ReleaseLevelHoldIfEligible and other callers of listIssueCommentsSince are untouched.

Open question (not a finding)

The correctness of reuse rests on the premise at issue_unpark_read_cache.go header: "GitHub moves updated_at on … a new, edited or deleted comment". New comments and label changes certainly bump it. I could not confirm from the tree or docs that editing or deleting an existing comment does. If it does not, a human who edits a typo into a valid /unpark-style command would wait for the 6 h TTL instead of the next pass. The backstop bounds the damage to a delay, so this is low severity either way, but it is the one assumption the tests cannot exercise against a real GitHub, and it would be worth a one-line note in the header if the author has seen it hold on the live hive.

Nothing else. The body's stated scope (reuse raw reads keyed by updated_at, no decision caching, in-memory only) matches the diff.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 38ac8ca into hivecommons:v6 Nov 8, 2026
67 of 68 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#17174813de01` sha256:e5a55ccb83f1520801845dfab6cd436f79b24056adec3a6523ee0165976aa0b6 β€” download from the dashboard or `hivectl review evidence #17174`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 813de01.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant