Skip to content

Re-measure cost per merged PR after GPT-6.1 Terra #14762

Closed

Description

@clubanderson

From the 2026-10-27 weekly competitor report, Top 5 item 5 (stay ahead): #14613 (comment)

GitHub added GPT-6.1 Terra to Copilot on 2026-10-26 (https://github.blog/changelog/2026-10-26-gpt-6-1-terra-in-github-copilot). A new model in the engines Hive drives can move Hive's cost per merged PR. Hosted vendors do not publish this number. Hive can, with intervals. The report's "$2.13 per PR" is the reviewer-only spend per hive-authored PR on one hive (Bluefin, Oct 13 to 27); the all-agent figure is higher. Both belong on one page with their definitions.

What to measure

  • Cost per merged PR, list price from token counts, all agents, per hive: hivecommons (hive's hive, v5), kubestellar/console (v6), projectbluefin (v6).
  • Median and 90% interval (5th to 95th percentile), not only the mean. Per-PR where attribution exists; per-day ratio where it does not.
  • Per author class of the PR being worked: hive (hivecommons-hive[bot]), human, other-bot (dependabot, renovate, mergeraptor, github-actions).
  • Before and after model availability. Baseline window: the 14 days before 2026-10-26. Comparison window: the first 14 days a hive routes any agent to GPT-6.1 Terra. Note which agent and lane changed model, and the price table date used for each window.
  • Reviewer spend alone, same splits, so the $2.13 figure can be refreshed with its definition attached.

Where the numbers come from

  • Dashboard GET /api/cost (daily_rollup[], price_table_date, merged_prs) and GET /api/cost/history on each hive. Pull in-pod with X-Hive-Internal: <dashboard-token>.
  • GET /api/launch-cost on each hive for per-PR attribution (by_ref[].share_usd). Today it retains about one day of launches (512 in the 2026-10-27 pull, truncated: true), so a 14-day per-PR series needs a daily pull or a longer retention window; see the note below.
  • Copilot session logs on each pod (/data/home/agents/<agent>/.copilot/session-state/<uuid>/events.jsonl, token counts from session.shutdown.modelMetrics) priced with src/pkg/tokens/pricing.go. This is the method behind the reviewer-spend split (extractor and attribution scripts exist; Andy has them).
  • Merged PRs from GitHub search: org:<org> is:pr is:merged author:app/hivecommons-hive merged:<from>..<to>.
  • Caveats to carry on the page: list price, not bills; no CI minutes; sessions without a shutdown record are priced $0 (floors); daily rollups are complete only from 2026-10-23 on the hosted hives.

Output

  • A page-ready markdown table with per-hive median, mean and 90% interval, author-class split, and the before/after columns, plus a CSV. First pass attached as a comment below. Publishing it as a public benchmark page is a separate decision.

Follow-up worth considering

  • /api/launch-cost retention long enough for a 14-day per-PR series, or a daily rollup of by_ref into /data/hive-metrics/, so the median and interval come from per-PR attribution rather than per-day ratios.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    kind/featureCategorizes issue or PR as related to a new feature.
    on Oct 28, 2026
  2. clubanderson commented on Oct 28, 2026

    @clubanderson
    MemberAuthor

    First measurement pass, 2026-10-27 about 20:20 EDT. Full write-up and CSV are on Andy's machine (~/hive-work/stay-ahead/cost-per-pr.{md,csv}); nothing published.

    1. All-agent spend per hive-authored merged PR

    Spend is the dashboard /api/cost daily_rollup on each hive. PRs are merged PRs authored by hivecommons-hive[bot] in that hive's org, from GitHub search. The daily rollups are complete only from 2026-10-23 (the Oct 22 repricing reset earlier days to about $0), so this series covers Oct 23 to 27, five days, not the 14 days the issue asks for. Median and interval are over daily spend / daily merged PRs, because per-PR attribution (/api/launch-cost) retains only about one day.

    Hive Org Days Spend (list) Merged PRs Mean $/PR Median daily $/PR 90% interval (daily)
    hive hivecommons 5 $5,006 396 $12.64 $13.11 $10.02 to $14.42
    console kubestellar 5 $579 42 $13.79 $16.60 $2.98 to $21.84
    bluefin projectbluefin 5 $1,560 90 $17.33 $20.39 $9.29 to $36.15
    all three $7,145 528 $13.53

    Console's Oct 25 rollup is $4.90 for 8 merged PRs, which looks like a recording gap; it drives the low end of console's interval.

    These match the 2026-10-27 blog preview (hive $12.51, console $12.10, bluefin $16.52 for Oct 23 to 27, computed a few hours earlier).

    2. Reviewer spend per PR, by author class

    Per-PR attribution from Copilot reviewer session logs on each pod (Oct 14 to 27), priced with src/pkg/tokens/pricing.go. Method and raw CSV: ~/hive-work/reviewer-spend/. Author class is the class of the PR's author, not the agent. Sessions without a shutdown record are priced $0, so these are floors. This is the series the report's "$2.13 per PR" belongs to: it is bluefin, hive-authored PRs, reviewer spend only ($2.16 here with a 14-day window).

    Hive Author class PRs Reviewer spend Mean $/PR Median $/PR 90% interval (per PR)
    hive hive 859 $3,855 $4.49 $2.84 $0.77 to $14.25
    hive human 445 $2,250 $5.06 $3.22 $0.59 to $15.97
    hive other-bot 16 $52 $3.27 $2.10 $0.84 to $9.58
    hive all 1763 $7,088 $4.02 $2.60 $0.51 to $13.53
    console hive 73 $110 $1.51 $1.40 $0.00 to $2.96
    console human 30 $58 $1.93 $1.07 $0.00 to $5.51
    console other-bot 9 $18 $2.01 $1.47 $0.80 to $5.05
    console all 131 $230 $1.76 $1.39 $0.00 to $4.38
    bluefin hive 148 $319 $2.16 $1.27 $0.56 to $7.90
    bluefin human 346 $989 $2.86 $1.95 $0.59 to $8.06
    bluefin other-bot 289 $428 $1.48 $0.80 $0.43 to $4.73
    bluefin all 807 $1,771 $2.19 $1.27 $0.46 to $6.96

    3. What is missing for the issue's target

    • Full 14-day all-agent series: needs daily rollups before Oct 23 (gone) or /api/launch-cost kept for 14 days.
    • Per-PR median and 90% interval for all-agent spend: needs /api/launch-cost by_ref retention or a daily snapshot. Today only reviewer spend has per-PR attribution.
    • Author-class split for all-agent spend: same dependency.
    • "After" column: no hive routes an agent to GPT-6.1 Terra yet. Re-run after the first 14 days on the new model, same method, and record the price table date for each window (hive and console 2026-10-26, bluefin 2026-10-22 for this pass).
  3. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    Scanner note on the per-PR retention caveat, not a claim (the measurement itself needs in-pod reads on three hives, which is owner-side):

    The truncated: true / 512 entries you saw on GET /api/launch-cost is the response default, not the stored window. parseHistoryQuery (src/pkg/dashboard/history_response.go) defaults max_points to 512 and accepts up to 10000, and the launch log ring (launchLogRingCap in src/pkg/tokens/launch.go) retains 4000 launch records. So GET /api/launch-cost?max_points=10000 (optionally with since= as a Unix-millisecond cutoff) returns the full retained per-PR series without any code change. Whether 4000 launches covers 14 days depends on the hive's kick rate; if it does not on the busiest hive, the daily by_ref rollup into /data/hive-metrics/ is the code follow-up worth filing, and I can take that as a child issue if you want it.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  4. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    🔒 Claimed by agent scanner (agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

  5. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    PR #14818 adds GET /api/launch-cost/history: a persisted daily per-ref rollup of launch attribution (day, then by_ref[] with ref, kind, launches, share_usd), kept ~2 years on the PVC and surviving restarts, so a 14-day per-PR series no longer depends on the ~1-day /api/launch-cost window. Existing fields are unchanged. It starts accumulating once deployed; author class is not recorded at launch time, so it is not in the rollup.

    The measurement run itself still needs an operator with access to all three hives (hivecommons, kubestellar/console, projectbluefin): pull /api/cost (daily_rollup), /api/launch-cost?max_points=10000 (and /api/launch-cost/history once deployed) and session logs from each, using the dashboard auth token header, then compute the median / 90% interval per author class before and after GPT-6.1 Terra. I can't reach the other hives' dashboards from this pod, so no numbers are produced here.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 5418e9933

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  6. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by agent scanner released (holding agent relaunched).

  7. github-actions commented on Oct 28, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions