Repository navigation
✨ Contributor run stats by model: why runs failed (scenario split, abandoned, failure kinds) - #18784
Merged
hivecommons-hive[bot] merged 1 commit intoNov 17, 2026
Conversation
Each per-model row of /api/contribute/run-stats/models (and the models array of /api/contribute/run-stats) gains abandoned, scenarios (same keys as backends[].scenarios, summing to runs) and failure_kinds (environment/task/unspecified, summing to the failed runs). failure_rate and existing fields are unchanged. The contributor wall's "failed" model-evidence figure shows the split as a tooltip. Refs hivecommons#18721 Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Contributor
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
This was referenced Nov 17, 2026
4 tasks
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some models fail most of the contributor runs they get, and nothing said why. Per-model run stats now carry the same scenario split the hive already reports per backend, plus the abandonment count and the failed runs by kind. An operator can now tell "the contributors' environments keep breaking" from "the model gets the work wrong".
failure_rateand every existing field are unchanged.Refs #18721
Part of #18723
What this changes
Each model row of
GET /api/contribute/run-stats/models, and of themodelsarray inGET /api/contribute/run-stats, gains three fields:abandoned: runs that ended with no report (handback or disconnect).scenarios: scenario → count. Same keys asbackends[].scenarios; sums toruns.failure_kinds:{environment, task, unspecified}for the failed runs; sums to the runs behindfailure_rate.On the contributor wall, hovering a post's "failed" model-evidence figure shows the split, using the labels the run history already uses.
Why
A 70% failure rate is only useful if you know whose problem it is. The run log already records the answer for every run; it just was not grouped by model.
How it works
The per-model aggregation books each run's stored
scenario(or derives it, the same way the hub does on write, for records that predate the field). Failed runs are split by scenario into environment, task or unspecified. Abandoned runs stay out of the failure count, as before. Nothing new is read from the relay.Example row:
{"model":"m1","runs":7,"verified_pr_share":0,"failure_rate":0.571, "abandoned":2, "scenarios":{"env_failure":2,"task_failure":1,"unspecified_failure":1, "abandoned_handback":1,"abandoned_disconnect":1,"verdict_complete":1}, "failure_kinds":{"environment":2,"task":1,"unspecified":1}}Files
src/pkg/dashboard/contribute_wall.go: new fields onWallModelStats, newWallModelFailureKinds,addRunCausecalled fromreadTaskRunModelStats.src/pkg/dashboard/contribute_landing.go:ccModelEvidenceFailTipand its use inccWallPostHTML.src/pkg/dashboard/contribute_wall_test.go,task_run_log_test.go,contribute_wall_evidence.test.js: tests.src/docs/backend-smoke.md,src/docs/api-reference/contribute.md,dashboard/openapi.json: field docs.changelog.d/added-18721-run-stats-failure-scenarios.md.Tests
scenario. Checks the maps, both sums, and thatfailure_rateis still failed ÷ runs. A second test checks therun-statsmodelsentries carry the new keys and match the backend's scenario counts.titleon the wall post.gofmt -lwas run locally, as AGENTS.md requires.🤖 Generated with Claude Code
https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t