Repository navigation
Closed
Labels
help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/featureCategorizes issue or PR as related to a new feature.Categorizes issue or PR as related to a new feature.
Description
From the 2026-10-27 weekly competitor report, Top 5 item 5 (stay ahead): #14613 (comment)
GitHub added GPT-6.1 Terra to Copilot on 2026-10-26 (https://github.blog/changelog/2026-10-26-gpt-6-1-terra-in-github-copilot). A new model in the engines Hive drives can move Hive's cost per merged PR. Hosted vendors do not publish this number. Hive can, with intervals. The report's "$2.13 per PR" is the reviewer-only spend per hive-authored PR on one hive (Bluefin, Oct 13 to 27); the all-agent figure is higher. Both belong on one page with their definitions.
What to measure
- Cost per merged PR, list price from token counts, all agents, per hive: hivecommons (hive's hive, v5), kubestellar/console (v6), projectbluefin (v6).
- Median and 90% interval (5th to 95th percentile), not only the mean. Per-PR where attribution exists; per-day ratio where it does not.
- Per author class of the PR being worked:
hive(hivecommons-hive[bot]),human,other-bot(dependabot, renovate, mergeraptor, github-actions). - Before and after model availability. Baseline window: the 14 days before 2026-10-26. Comparison window: the first 14 days a hive routes any agent to GPT-6.1 Terra. Note which agent and lane changed model, and the price table date used for each window.
- Reviewer spend alone, same splits, so the $2.13 figure can be refreshed with its definition attached.
Where the numbers come from
- Dashboard
GET /api/cost(daily_rollup[],price_table_date,merged_prs) andGET /api/cost/historyon each hive. Pull in-pod withX-Hive-Internal: <dashboard-token>. GET /api/launch-coston each hive for per-PR attribution (by_ref[].share_usd). Today it retains about one day of launches (512 in the 2026-10-27 pull,truncated: true), so a 14-day per-PR series needs a daily pull or a longer retention window; see the note below.- Copilot session logs on each pod (
/data/home/agents/<agent>/.copilot/session-state/<uuid>/events.jsonl, token counts fromsession.shutdown.modelMetrics) priced withsrc/pkg/tokens/pricing.go. This is the method behind the reviewer-spend split (extractor and attribution scripts exist; Andy has them). - Merged PRs from GitHub search:
org:<org> is:pr is:merged author:app/hivecommons-hive merged:<from>..<to>. - Caveats to carry on the page: list price, not bills; no CI minutes; sessions without a shutdown record are priced $0 (floors); daily rollups are complete only from 2026-10-23 on the hosted hives.
Output
- A page-ready markdown table with per-hive median, mean and 90% interval, author-class split, and the before/after columns, plus a CSV. First pass attached as a comment below. Publishing it as a public benchmark page is a separate decision.
Follow-up worth considering
/api/launch-costretention long enough for a 14-day per-PR series, or a daily rollup ofby_refinto/data/hive-metrics/, so the median and interval come from per-PR attribution rather than per-day ratios.
Metadata
Metadata
Assignees
Labels
help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/featureCategorizes issue or PR as related to a new feature.Categorizes issue or PR as related to a new feature.
Activity
clubanderson commentedon Oct 28, 2026
First measurement pass, 2026-10-27 about 20:20 EDT. Full write-up and CSV are on Andy's machine (
~/hive-work/stay-ahead/cost-per-pr.{md,csv}); nothing published.1. All-agent spend per hive-authored merged PR
Spend is the dashboard
/api/costdaily_rollupon each hive. PRs are merged PRs authored byhivecommons-hive[bot]in that hive's org, from GitHub search. The daily rollups are complete only from 2026-10-23 (the Oct 22 repricing reset earlier days to about $0), so this series covers Oct 23 to 27, five days, not the 14 days the issue asks for. Median and interval are over daily spend / daily merged PRs, because per-PR attribution (/api/launch-cost) retains only about one day.Console's Oct 25 rollup is $4.90 for 8 merged PRs, which looks like a recording gap; it drives the low end of console's interval.
These match the 2026-10-27 blog preview (hive $12.51, console $12.10, bluefin $16.52 for Oct 23 to 27, computed a few hours earlier).
2. Reviewer spend per PR, by author class
Per-PR attribution from Copilot reviewer session logs on each pod (Oct 14 to 27), priced with
src/pkg/tokens/pricing.go. Method and raw CSV:~/hive-work/reviewer-spend/. Author class is the class of the PR's author, not the agent. Sessions without a shutdown record are priced $0, so these are floors. This is the series the report's "$2.13 per PR" belongs to: it is bluefin, hive-authored PRs, reviewer spend only ($2.16 here with a 14-day window).3. What is missing for the issue's target
/api/launch-costkept for 14 days./api/launch-costby_refretention or a daily snapshot. Today only reviewer spend has per-PR attribution.hivecommons-hive commentedon Oct 28, 2026
Scanner note on the per-PR retention caveat, not a claim (the measurement itself needs in-pod reads on three hives, which is owner-side):
The
truncated: true/ 512 entries you saw onGET /api/launch-costis the response default, not the stored window.parseHistoryQuery(src/pkg/dashboard/history_response.go) defaultsmax_pointsto 512 and accepts up to 10000, and the launch log ring (launchLogRingCapinsrc/pkg/tokens/launch.go) retains 4000 launch records. SoGET /api/launch-cost?max_points=10000(optionally withsince=as a Unix-millisecond cutoff) returns the full retained per-PR series without any code change. Whether 4000 launches covers 14 days depends on the hive's kick rate; if it does not on the busiest hive, the dailyby_refrollup into/data/hive-metrics/is the code follow-up worth filing, and I can take that as a child issue if you want it.🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:unknown— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
hivecommons-hive commentedon Oct 28, 2026
🔒 Claimed by agent
scanner(agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply/unclaimto release, or/claimto take it over if you outrank the holder.hivecommons-hive commentedon Oct 28, 2026
PR #14818 adds
GET /api/launch-cost/history: a persisted daily per-ref rollup of launch attribution (day, thenby_ref[]withref,kind,launches,share_usd), kept ~2 years on the PVC and surviving restarts, so a 14-day per-PR series no longer depends on the ~1-day/api/launch-costwindow. Existing fields are unchanged. It starts accumulating once deployed; author class is not recorded at launch time, so it is not in the rollup.The measurement run itself still needs an operator with access to all three hives (hivecommons, kubestellar/console, projectbluefin): pull
/api/cost(daily_rollup),/api/launch-cost?max_points=10000(and/api/launch-cost/historyonce deployed) and session logs from each, using the dashboard auth token header, then compute the median / 90% interval per author class before and after GPT-6.1 Terra. I can't reach the other hives' dashboards from this pod, so no numbers are produced here.🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:5418e9933— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
hivecommons-hive commentedon Oct 28, 2026
🔓 Claim by agent
scannerreleased (holding agent relaunched).🐛 dashboard: document GET /api/launch-cost/history in openapi.json
✨ dashboard: keep a daily per-PR launch-cost rollup for cost-per-merg…
github-actions commentedon Oct 28, 2026
@clubanderson this issue was closed by @hivecommons-hive[bot] as completed.
If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.
If you are still seeing the problem, comment
/reopenand it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.