Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-pr-models-attributed-history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The Governor's **PRs by model** card and Operations' **Most effective models** table now count every attributed PR in their `30d` and `all` windows, not only those the closed-PR scan touched in the last 14 days. That scan stops at PRs updated in the last `HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK` (default 14 days) and kept its results only in memory, so a PR closed and left alone for longer silently dropped out and a restart lost them all. Models lost different shares that way (on one hive 55% of one model's PRs since late October and 90% of another's), so the ranking compared unequal samples. The hive now keeps a history of attributed PRs in `/data/attributed-pr-history.json` (parsed attribution, outcome and rework fields only, never PR bodies) that survives restarts, and a one-time backfill fills it back 30 days (`HIVE_ATTRIBUTED_PR_BACKFILL_LOOKBACK`). The backfill spends at most two list pages and five rework measurements per enumeration pass, only while the GitHub API budget is normal, resumes where it stopped after a restart and makes no calls once done; the per-pass closed-PR scan stays bounded as before. The `coverage` block reports where the history starts (new `closed_prs_since`) and marks `closed_prs_partial` only when a window starts before it.
4 changes: 4 additions & 0 deletions src/cmd/hive/github_client_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,10 @@ func (b *boot) applyGitHubClientConfigHooks(client *github.Client) {
// Evaluated heads persist so a restart or rebuild neither re-diffs every
// open PR nor blanks the Matrix feed until the next sweep (#16276).
client.SetSentinelHeadsPath(sentinelHeadsStatePath)
// Attributed PRs persist past the bounded closed-PR scan and a restart,
// so the PRs-by-model 30d/All windows rest on the same sample for every
// model; the store also carries the one-time backfill's cursor.
client.SetAttributedPRHistoryPath(github.AttributedPRHistoryPath)
client.SetAPIBudgetThresholds(b.cfg.GitHub.APIReserve, b.cfg.GitHub.APICritical)
// The conditional-request cache is process-wide, not per client; size it
// from github.etag_cache_max_mb / _max_entries so a hive that evicts
Expand Down
5 changes: 3 additions & 2 deletions src/docs/admin-mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -488,8 +488,9 @@ capped at 256 KiB.

Other notes:
- `effective_models_read` also returns `coverage`. It says how far back the
closed-PR scan reaches and whether PR or run history is partial for the
window, so an `all` answer is not mistaken for lifetime totals.
hive's stored attributed PR history is complete (`closed_prs_since`) and
whether PR or run history is partial for the window, so an `all` answer
is not mistaken for lifetime totals.
- None of them carries cost: `spend_read` has spend by model.

**Meta tools**
Expand Down
2 changes: 1 addition & 1 deletion src/docs/api-reference/agents-and-controls.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f
| `POST` | `/api/unpin/{agent}/{dimension}` | Owner only | Unpin | `pkg/dashboard/api.go:149` |
| `POST` | `/api/restart/{agent}` | Owner only | Restart | `pkg/dashboard/api.go:153` |
| `GET` | `/api/model-advisor` | Dashboard auth/session | Model Advisor | `pkg/dashboard/api.go:171` |
| `GET` | `/api/governor/pr-models` | Dashboard auth/session | Agent-authored PR distribution by normalized attribution model/backend for `window=7d`, `30d`, or `all`, including per-model rework stats, Wilson 95% `*_ci` intervals on the rates (omitted when the denominator is 0), a `coverage` block (closed-PR lookback and run-log partiality, as on `/api/contribute/effective-models`), `earliest_pr_created_at`, the top 10 most-reworked PRs, and per-bucket `hive_cost` (`usd`, `launches`, `prs_with_cost`, `usd_per_merged_pr`: the hive's estimated review/fix spend on that model's PRs from the cached `/api/launch-cost` snapshot; `usd` is `null`, not `0`, when nothing was attributed; omitted while the snapshot is not ready) with a top-level `hive_cost_coverage` (`ready`, `window_hours`, `unattributed_usd`, `backend_unsupported_usd`, `disclaimer`). Optional `model=<id>` (matched after `modelfamily.Canonical`, so raw spellings such as `anthropic/claude-opus-5-5` work) adds `prs[]`: every PR in the window in that model's bucket, with the `most_reworked` fields plus `outcome` (`merged` / `closed_unmerged` / `open`), `agent`, `created_at`, `merged_at` and `first_pass`, sorted by review rounds + fix attempts then newest, capped at 200 with `prs_truncated: true` past that. An unknown model returns `prs: []`; without `model` the response is unchanged | `pkg/dashboard/api.go:172` |
| `GET` | `/api/governor/pr-models` | Dashboard auth/session | Agent-authored PR distribution by normalized attribution model/backend for `window=7d`, `30d`, or `all`, including per-model rework stats, Wilson 95% `*_ci` intervals on the rates (omitted when the denominator is 0), a `coverage` block (how far back the stored attributed PR history is complete, `closed_prs_since`, and closed-PR and run-log partiality, as on `/api/contribute/effective-models`), `earliest_pr_created_at`, the top 10 most-reworked PRs, and per-bucket `hive_cost` (`usd`, `launches`, `prs_with_cost`, `usd_per_merged_pr`: the hive's estimated review/fix spend on that model's PRs from the cached `/api/launch-cost` snapshot; `usd` is `null`, not `0`, when nothing was attributed; omitted while the snapshot is not ready) with a top-level `hive_cost_coverage` (`ready`, `window_hours`, `unattributed_usd`, `backend_unsupported_usd`, `disclaimer`). Optional `model=<id>` (matched after `modelfamily.Canonical`, so raw spellings such as `anthropic/claude-opus-5-5` work) adds `prs[]`: every PR in the window in that model's bucket, with the `most_reworked` fields plus `outcome` (`merged` / `closed_unmerged` / `open`), `agent`, `created_at`, `merged_at` and `first_pass`, sorted by review rounds + fix attempts then newest, capped at 200 with `prs_truncated: true` past that. An unknown model returns `prs: []`; without `model` the response is unchanged | `pkg/dashboard/api.go:172` |
| `GET` | `/api/reviewer/accuracy` | Dashboard auth/session | Reviewer verdict calibration for the configured window: false-approve/false-block rates and confidence buckets per perspective and reviewer model | `pkg/dashboard/api.go:239` |
| `GET` | `/api/review/queue` | Dashboard auth/session | PR review queue: every open PR (agent and contributor, actionable and held) ranked by triage class, confidence band, CI state and age, with per-PR `reasons`; unreviewed items carry `pending_since` and `redispatch_count` from the review dispatch state; each entry carries `stage` `{stage, entered_at, last_activity_at, quiet_for, stalled, waiting_on_hive, past_dwell, history[]}` from the stage-label store (`stalled`/`past_dwell` follow `governor.labels.stage.dwell` and `stalled.require_quiet`; `history[]` is `{from, to, at, reason, regressed}`, newest last, capped at 8) and the top-level `hive_held` counts PRs past their stage dwell while waiting on the hive (`waiting_on_hive` and `past_dwell` both true); paged with `limit` (1-200, default 50) and `offset` | `pkg/dashboard/api.go:533` |
| `POST` | `/api/review/queue/{owner}/{repo}/{number}/superseded/close` | Owner only | Close a superseded PR now (grace window or not): posts the close comment, adds `hive/superseded`, closes it | `pkg/dashboard/api.go:536` |
Expand Down
2 changes: 1 addition & 1 deletion src/docs/contributor-relay.md
Original file line number Diff line number Diff line change
Expand Up @@ -1233,7 +1233,7 @@ After a contributor completes an issue, the hub keeps that issue out of the queu

### Queue hold and priority

The **Operations** tab also includes a public-safe, collapsible **Most effective models** panel, collapsed by default so the work, runs, decisions and queue cards sit near the top; its header still shows the ranked-count summary once the panel has been opened. It reads the same aggregate PR rework data as the Governor PRs-by-model view plus the contributor task-run log, then ranks model + CLI pairs for `7d`, `30d`, or `all` by visible measures: merged PRs, first-pass merge rate, average review rounds, average fix attempts, verified-PR run rate, failure rate, and completed-without-PR (“nothing to ship”) rate. Rows below `HIVE_CONTRIBUTE_EFFECTIVE_MODELS_MIN_PRS` merged PRs (default `5`) are shown under “Not enough data yet” instead of being ranked, and the response is aggregate-only: no contributor usernames, pane output, tokens, or per-contributor breakdowns. The panel only requests `/api/contribute/effective-models` once it is expanded, and the choice is remembered in browser localStorage in both directions (`1` collapsed, `0` expanded; no stored value means the collapsed default). Run-derived rates show `—` for rows with no recorded runs (they come from the contributor task-run log only), and the runtime label reflects where inference runs: a Pi row is `local` only for a local provider such as `lemonade/` or `ollama`, `hosted` for a hosted provider prefix such as `openai-codex/`, and `unknown` otherwise. The response's `coverage` object states the closed-PR scan lookback (`HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK`, default 14 days) and the oldest retained run (live log plus its rotated `.1` file), and flags `30d`/`all` windows that reach past retained history as partial; the panel shows that coverage under the table.
The **Operations** tab also includes a public-safe, collapsible **Most effective models** panel, collapsed by default so the work, runs, decisions and queue cards sit near the top; its header still shows the ranked-count summary once the panel has been opened. It reads the same aggregate PR rework data as the Governor PRs-by-model view plus the contributor task-run log, then ranks model + CLI pairs for `7d`, `30d`, or `all` by visible measures: merged PRs, first-pass merge rate, average review rounds, average fix attempts, verified-PR run rate, failure rate, and completed-without-PR (“nothing to ship”) rate. Rows below `HIVE_CONTRIBUTE_EFFECTIVE_MODELS_MIN_PRS` merged PRs (default `5`) are shown under “Not enough data yet” instead of being ranked, and the response is aggregate-only: no contributor usernames, pane output, tokens, or per-contributor breakdowns. The panel only requests `/api/contribute/effective-models` once it is expanded, and the choice is remembered in browser localStorage in both directions (`1` collapsed, `0` expanded; no stored value means the collapsed default). Run-derived rates show `—` for rows with no recorded runs (they come from the contributor task-run log only), and the runtime label reflects where inference runs: a Pi row is `local` only for a local provider such as `lemonade/` or `ollama`, `hosted` for a hosted provider prefix such as `openai-codex/`, and `unknown` otherwise. Closed PRs come from the hive's attributed PR history (`/data/attributed-pr-history.json`), which keeps every attributed PR the enumeration has seen after the bounded live scan (`HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK`, default 14 days) stops listing it, and which a one-time backfill fills back to `HIVE_ATTRIBUTED_PR_BACKFILL_LOOKBACK` (default 30 days). The response's `coverage` object states how far back that history is complete (`closed_prs_since` and `closed_pr_lookback_days`) and the oldest retained run (live log plus its rotated `.1` file), and flags windows that start before retained history as partial: `30d` until the backfill finishes, `all` unless the history reaches back to a repo's first PR. The panel shows that coverage under the table.

The dashboard **Governor** card's **PRs by model** section uses that same model-effectiveness aggregation for the same `7d`, `30d`, and `all` windows. It keeps the merged/open/closed bar for PR volume, adds compact columns for merged count, first-pass merge rate, verified-PR run rate, failure rate, and “nothing to ship” rate, and badges ranked models that clear the merged-PR threshold. Operators can toggle the row order between effectiveness rank (default) and raw PR count.

Expand Down
3 changes: 2 additions & 1 deletion src/docs/env-vars.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ behavior. [Maintenance limitations](#keeping-this-reference-current) still apply
| `HIVE_SCANNER_BUSY_BRANCH_MINUTES` | No | `60` | Busy-branch window for the repair queues ([#18488](https://github.com/hivecommons/hive/issues/18488)). A red or conflicted PR the hive App did not open is listed as "suggest, do not push" instead of as a push target while its head is younger than this many minutes or a claim by another worker covers the PR or a linked issue; the agent posts the fix as one suggestion comment for the worker to apply as its own signed commit. Whole minutes; unset, unparsable or non-positive values keep `60`. |
| `HIVE_RELEASE_FEED_INTERVAL` | No | `governor.release_feeds.interval_s`, else `15m` | Poll interval of the release-feed watcher ([#18494](https://github.com/hivecommons/hive/issues/18494)) as a Go duration. An explicit `interval_s` wins over it; never below `1m`. Invalid or non-positive values fall back to `15m`. See [release-feeds.md](release-feeds.md). |
| `HIVE_RELEASE_FEED_COOLDOWN` | No | `governor.release_feeds.cooldown_s` / per-agent `cooldown_s`, else `1h` | Default minimum gap between two release-feed kicks of one agent, as a Go duration. The YAML keys win over it. See [release-feeds.md](release-feeds.md#cooldown). |
| `HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK` | No | `14d` | Bounds the per-repo closed-PR attribution scan that feeds dashboard outcome/rework metrics during PR enumeration. The scan lists closed PRs sorted by `updated_at` descending, stops when results are older than this window, and also stops after five pages so large repositories are never walked from history on every governor tick. Values accept Go durations such as `336h`, or day counts such as `14`/`14d`; invalid or non-positive values fall back to `14d`. |
| `HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK` | No | `14d` | Bounds the per-repo closed-PR attribution scan that feeds dashboard outcome/rework metrics during PR enumeration. The scan lists closed PRs sorted by `updated_at` descending, stops when results are older than this window, and also stops after five pages so large repositories are never walked from history on every governor tick. Values accept Go durations such as `336h`, or day counts such as `14`/`14d`; invalid or non-positive values fall back to `14d`. Each scan's attributed PRs are also written to the attributed PR history (`/data/attributed-pr-history.json`), so a PR stays in the PRs-by-model and Most effective models windows after this scan stops listing it, and across restarts. |
| `HIVE_ATTRIBUTED_PR_BACKFILL_LOOKBACK` | No | `30d` | How far back, by PR creation, the one-time backfill of the attributed PR history walks each repo's closed PRs (newest first). It runs once per repo when the history is new, spends at most two list pages and five rework measurements per enumeration pass across all repos and only while the GitHub API budget is normal, resumes from its saved cursor after a restart, and makes no calls once complete. Until it finishes, windows that reach past the live scan's lookback report `closed_prs_partial`. Same value syntax as `HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK`. |
| `HIVE_LEVEL` | No | config/pack value | ACMM level bootstrap/override used by hosted flows and the entrypoint pack selection. |
| `HIVE_ID` | No | config or generated id | Stable hive/spoke identifier override; passed through to launched agents. |
| `HIVE_CLUSTER_ID` | No | config or hub-provisioned value | Hosted cluster identifier override. |
Expand Down
2 changes: 1 addition & 1 deletion src/docs/scale-envelope.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ ones to:
2. **Truncation must preserve the prioritization signal, not cut an arbitrary slice.**
This holds today and is easy to assume backwards:
- Issues are sorted by actionable rank tier, then **oldest first** within each tier before any cap is applied
(`RankActionableIssues`, `src/pkg/github/client.go:5652-5662`, descending `AgeMinutes` within each tier).
(`RankActionableIssues`, `src/pkg/github/client.go:5686-5696`, descending `AgeMinutes` within each tier).
- PRs are sorted by **review class then oldest-first within class**
(`SortPullRequestsForReview`, `src/pkg/github/review_priority.go:334`;
fixes → refactors/docs → tests).
Expand Down
2 changes: 1 addition & 1 deletion src/pkg/adminmcp/tools_model_effectiveness.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ const prModelsReadDescription = "Read Governor → \"PRs by model\" through GET

const contributorModelStatsReadDescription = "Read the contributor run stats by model through GET /api/contribute/run-stats/models: per model, runs, verified_pr_share (runs that ended in a verified PR) and failure_rate, over window_days. days is 0-365 (default 7); 0 reads every recorded run. Rows are sorted by model name. limit caps the models rows and offset skips rows; a capped answer says so in _admin_mcp and gives next_offset. Read-only."

const effectiveModelsReadDescription = "Read the Operations page's \"Most effective models\" table through GET /api/contribute/effective-models, the endpoint the panel reads. window is 7d (default), 30d or all. filter is all (default), hive (PRs the hive's own agents opened; these have no contributor runs, so run columns are empty) or contributor (only PRs a verified contributor run produced, with those runs). Returns window, filter, min_merged_prs, ranked (models with at least min_merged_prs merged PRs, best score first, with effectiveness_rank), insufficient (the rest) and coverage (closed_pr_lookback_days, closed_prs_partial, runs_since, runs_partial: how much history the window really covers). Each row has model, backends, sources, runtime, score, prs, merged_prs, first-pass merges and rate, average review rounds and fix attempts, runs, and verified-PR, failed and nothing-to-ship run counts and rates. ranked is returned whole; limit caps the insufficient rows and offset skips rows of it. Read-only."
const effectiveModelsReadDescription = "Read the Operations page's \"Most effective models\" table through GET /api/contribute/effective-models, the endpoint the panel reads. window is 7d (default), 30d or all. filter is all (default), hive (PRs the hive's own agents opened; these have no contributor runs, so run columns are empty) or contributor (only PRs a verified contributor run produced, with those runs). Returns window, filter, min_merged_prs, ranked (models with at least min_merged_prs merged PRs, best score first, with effectiveness_rank), insufficient (the rest) and coverage (closed_pr_lookback_days, closed_prs_partial, closed_prs_since, runs_since, runs_partial: how much history the window really covers; closed_prs_since is when the hive's stored PR history starts). Each row has model, backends, sources, runtime, score, prs, merged_prs, first-pass merges and rate, average review rounds and fix attempts, runs, and verified-PR, failed and nothing-to-ship run counts and rates. ranked is returned whole; limit caps the insufficient rows and offset skips rows of it. Read-only."

// ModelEffectivenessTool reports whether tool is one of the model
// effectiveness reads.
Expand Down
Loading
Loading