Skip to content

πŸ› pr-models: keep attributed PR history past the 14-day closed-PR scan - #19371

Merged
hivecommons-hive[bot] merged 6 commits into
hivecommons:v6from
Danathar:danathar/pr-models-attributed-history
Nov 20, 2026
Merged

hivecommons-hive[bot] merged 6 commits into
hivecommons:v6from
Danathar:danathar/pr-models-attributed-history

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

Governor "PRs by model" (/api/governor/pr-models) and Operations "Most effective models" (/api/contribute/effective-models) offer 30d and "all" windows. But closed attributed PRs came only from a live per-repo scan that stops at PRs updated in the last 14 days, and the results lived only in memory. A PR closed and left alone for more than 14 days dropped out, and a restart lost everything.

That skews the ranking, not just the history, because models lose different shares depending on when they were used. On projectbluefin (707 trailer-carrying PRs since 2026-10-20), the cutoff dropped 55% of claude-opus-5.5's PRs, 87% of gpt-6.1-sol's and 90% of claude-fable-5.1's, or 52% overall. The coverage block admitted the gap (closed_prs_partial: true), but the ranking was still computed on unequal samples.

This keeps a persistent history of attributed PRs and fills it back 30 days once.

What changes

  • Stored history. /data/attributed-pr-history.json holds one record per attributed PR: repo, number, title, URL, labels, agent, backend, model, effort, state, created/closed/merged times, supersedes and the rework stats the cards read. It never holds PR bodies. A record is written when the hive first sees a PR and updated when it closes. It survives restarts, is written at most once per enumeration pass (only when something changed), and is pruned at 400 days or 20,000 closed records. It follows the PR rework cache in the same package: one versioned JSON file, written tmp+rename under a mutex.
  • Both endpoints read it. Live attributed PRs are merged with the stored closed ones, and live wins. Response shapes are unchanged, and so are the scoring formula and min_merged_prs.
  • One-time backfill back 30 days (HIVE_ATTRIBUTED_PR_BACKFILL_LOOKBACK). It runs as one bounded step per enumeration pass:
    • At most 2 closed-PR list pages and 5 rework measurements across all repos.
    • It runs only while the API budget is normal, at low pacing priority.
    • Only one step runs at a time.
    • It resumes from a cursor saved per repo, and re-reads the last page so a reopened PR can't shift one past it.
    • Rate limits, pacing shed or a timed-out pass stop the step without costing anything. Repeated failures for a PR or a listing are capped.
    • A finished repo makes no calls, so once the backfill is done, per-pass GitHub calls are the same as before.
  • Coverage tells the truth.
    • It is measured in PR creation time, because that is what the windows filter on.
    • closed_prs_since (new) is where the history really starts. closed_prs_partial is true only when the requested window starts before it.
    • A repo that has enumerated but never had a closed scan counts as uncovered, and so does a repo whose last closed scan is older than the live lookback. So the flag can't claim a window is complete while PRs are missing.
    • A hive with no store reports exactly what it does today.

Kept as before

Known limits

  • Backfill pages are walked by number. More than 100 PRs reopened between two passes of one walk could still skip one. This is documented in code.
  • The backfill and the live rework enrichment can overlap on the rework cache file, and the last writer wins. Measurements are also kept on the history records, so nothing is lost there.
  • Prometheus metrics still use live data only.

Tests

  • Store (github):
    • a PR closed 20 days ago stays after the 14-day scan drops it, survives a reload, and no body lands in the file;
    • an unwritable store doesn't fail enumeration;
    • a null record in the file is dropped;
    • records already pruned are not re-inserted;
    • retention and stale-open pruning.
  • Backfill:
    • stops at its page budget, resumes from the saved cursor after a reload, finishes, then makes no calls;
    • makes no calls in conserve mode;
    • rework tries count only per-PR errors, and transient errors have their own cap;
    • a listing error moves on to the next repo, and repeated listing failures stop the walk without claiming coverage.
  • Coverage:
    • continuity across a scan gap;
    • a never-enumerated repo doesn't count;
    • an enumerated but never-scanned repo is uncovered;
    • a stale scan is uncovered.
  • Dashboard:
    • a 30d window counts PRs closed 3, 16 and 25 days ago, and the live+store merge doesn't double count;
    • coverage is partial or not for 30d and all, with the backfill done or running, for both endpoints;
    • with no store, coverage is unchanged.

Nothing was run locally (AGENTS.md); CI is the first build.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

Danathar and others added 5 commits November 19, 2026 23:25
…t every PR

The Governor "PRs by model" card and Operations "Most effective models"
table took closed attributed PRs only from the live per-repo scan, which
stops at PRs updated in the last 14 days and keeps its result in memory.
A PR closed and left alone for longer dropped out of the 30d/All windows,
a restart lost everything, and models lost different shares depending on
when they were used (projectbluefin: 52% of 707 PRs since 10-20 lost,
from 55% of claude-opus-5.5 to 90% of claude-fable-5.1), so the ranking
compared unequal samples.

- pkg/github/attributed_pr_history.go: a JSON store at
  /data/attributed-pr-history.json (tmp+rename, like the PR rework cache)
  of every attributed PR the enumeration sees, open or closed: parsed
  trailer fields, state, times, labels, supersedes and rework stats, never
  a body. Written once per enumeration pass; closed records older than 400
  days or beyond 20000 are pruned and coverage moves forward to match.
- A one-time backfill walks closed PRs by creation, newest first, back 30
  days (HIVE_ATTRIBUTED_PR_BACKFILL_LOOKBACK) and measures rework for
  stored closed PRs lacking it. At most 2 pages and 5 rework measurements
  per pass across repos, only in the normal API budget mode, cursor saved
  in the store, marked done per repo; a done repo costs no calls.
- The live closed scan is unchanged (14-day lookback, 5-page cap); it now
  also records its floor so the store tracks whether scans have been
  continuous. Coverage start = the first floor of the current unbroken run
  of scans, or the backfill horizon once the backfill finished within that
  run, or full history when a list ran out.
- Both endpoints merge the store's closed PRs under the live records and
  report coverage from it: new closed_prs_since, closed_pr_lookback_days
  as the days covered, closed_prs_partial only when the window starts
  before the coverage start (All: unless full). Scoring and min_merged_prs
  are unchanged.
- Store and backfill errors only warn; enumeration never fails on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
- Rename the backfill test fake (collided with backfillTestServer in
  review_backlog_split_backfill_test.go).
- Rework tries count only errors about the PR itself. Rate limit, pacing
  shed and a cancelled or expired pass end the pass; 5xx/network move on
  to the next repo; neither costs a try.
- A listing error that is not about the budget keeps that repo's cursor
  for the next pass and lets the pass go on to the next repo.
- upsert skips closed PRs created before the retention cutoff or
  pruned_through, so they are not re-inserted and pruned (and the file
  rewritten) every pass.
- Single-flight the backfill step with a TryLock: several callers run
  EnumerateActionable and would otherwise list the same pages twice.
- Repos that never scanned are left out of the coverage start instead of
  making every window partial; with no scanned repo nothing is covered.
- Null entries in the file are dropped on load (they panicked in the
  walks inside enumeration).
- Open records carry last_seen_at (rewritten at most daily) and are
  pruned when not seen within the retention; records of repos no longer
  configured are outside the record cap and age out by retention.
- enrichAttributedPRRework returns its cache so the history does not read
  the rework cache file a second time per repo.
- A resumed backfill walk re-reads the page it read last, catching up to
  a page of PRs shifted behind the cursor by reopens; the remaining case
  is documented.
- docs/admin-mcp.md: coverage now describes the stored history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
- A rework fetch lost to a GitHub hiccup (5xx, network, timeout) now
  counts against its own cap of 10 and the next PR gets its turn; before,
  the fixed pending order put the same failing PR first every pass and
  the repo's backfill could never finish. Stop-class errors still end
  the pass without counting.
- A repo whose live enumeration succeeded in this process but has no
  closed-scan state is uncovered (its closed PRs are missing while its
  open ones are live); only repos that never enumerated are left out.
- A repo whose last closed scan is older than the live lookback is
  uncovered: PRs closed since are neither stored nor live.
- Failed backfill listings (per-repo errors or hiccups) are counted per
  walk and reset by a good page; after 5 the walk stops, the repo stays
  not done, and coverage keeps reporting that it does not reach the
  horizon.
- Note the rework cache's last-writer-wins overlap in a comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
…ibuted-history

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 20, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#19368 πŸ› fix: persist attributed closed PRs so 30d windows cover 30 days CONFLICT src/pkg/dashboard/api_governor_pr_models.go (conflict)
src/pkg/dashboard/contribute_effective_models.go (conflict)
src/pkg/github/client.go (conflict)

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 20, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • check-failure

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (2 waiting): fixes are handed out before new work as soon as you have capacity.

…scan's

The test expected model "auto", but the trailer's model=auto normalizes
(NormalizeAttributionModel) to "unknown" for live and stored PRs alike.
Compare against what the live scan parsed instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
@hivecommons-hive
hivecommons-hive Bot merged commit 2d427cb into hivecommons:v6 Nov 20, 2026
72 of 73 checks passed
@hivecommons-hive

hivecommons-hive Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#19371e76d133` sha256:30e51162bb297517db799e5c0bc0d93091cdefbffcbdc3ecd4b9e05eff018b40 β€” download from the dashboard or `hivectl review evidence #19371`.

merged by Hive trusted-author auto-merge: author Danathar holds role owner and GitHub permission maintain on repo; CI green at e76d133.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant