Skip to content

✨ Contributor run stats by model: why runs failed (scenario split, abandoned, failure kinds) - #18784

Merged
hivecommons-hive[bot] merged 1 commit into
hivecommons:v5from
Danathar:feat/run-stats-failure-scenarios
Nov 17, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
hivecommons:v5from
Danathar:feat/run-stats-failure-scenarios

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

Some models fail most of the contributor runs they get, and nothing said why. Per-model run stats now carry the same scenario split the hive already reports per backend, plus the abandonment count and the failed runs by kind. An operator can now tell "the contributors' environments keep breaking" from "the model gets the work wrong". failure_rate and every existing field are unchanged.

Refs #18721
Part of #18723

What this changes

Each model row of GET /api/contribute/run-stats/models, and of the models array in GET /api/contribute/run-stats, gains three fields:

  • abandoned: runs that ended with no report (handback or disconnect).
  • scenarios: scenario → count. Same keys as backends[].scenarios; sums to runs.
  • failure_kinds: {environment, task, unspecified} for the failed runs; sums to the runs behind failure_rate.

On the contributor wall, hovering a post's "failed" model-evidence figure shows the split, using the labels the run history already uses.

Why

A 70% failure rate is only useful if you know whose problem it is. The run log already records the answer for every run; it just was not grouped by model.

How it works

The per-model aggregation books each run's stored scenario (or derives it, the same way the hub does on write, for records that predate the field). Failed runs are split by scenario into environment, task or unspecified. Abandoned runs stay out of the failure count, as before. Nothing new is read from the relay.

Example row:

{"model":"m1","runs":7,"verified_pr_share":0,"failure_rate":0.571,
 "abandoned":2,
 "scenarios":{"env_failure":2,"task_failure":1,"unspecified_failure":1,
              "abandoned_handback":1,"abandoned_disconnect":1,"verdict_complete":1},
 "failure_kinds":{"environment":2,"task":1,"unspecified":1}}

Files

  • src/pkg/dashboard/contribute_wall.go: new fields on WallModelStats, new WallModelFailureKinds, addRunCause called from readTaskRunModelStats.
  • src/pkg/dashboard/contribute_landing.go: ccModelEvidenceFailTip and its use in ccWallPostHTML.
  • src/pkg/dashboard/contribute_wall_test.go, task_run_log_test.go, contribute_wall_evidence.test.js: tests.
  • src/docs/backend-smoke.md, src/docs/api-reference/contribute.md, dashboard/openapi.json: field docs.
  • changelog.d/added-18721-run-stats-failure-scenarios.md.

Tests

  • Go: one model with an env failure, a task failure, a failure with no kind, a handback, a disconnect, a verdict completion and one legacy record with no scenario. Checks the maps, both sums, and that failure_rate is still failed ÷ runs. A second test checks the run-stats models entries carry the new keys and match the backend's scenario counts.
  • JS: the tooltip text, unknown scenarios, an older hub with no split, and the escaped title on the wall post.
  • CI: this PR's own v2 Tests and v2 CI checks. Only gofmt -l was run locally, as AGENTS.md requires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t

Each per-model row of /api/contribute/run-stats/models (and the models
array of /api/contribute/run-stats) gains abandoned, scenarios (same keys
as backends[].scenarios, summing to runs) and failure_kinds
(environment/task/unspecified, summing to the failed runs). failure_rate
and existing fields are unchanged. The contributor wall's "failed"
model-evidence figure shows the split as a tooltip.

Refs hivecommons#18721

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18781 ✨ Spend and contributor run stats by model: add the canonical model name CONFLICT dashboard/openapi.json (conflict)
src/docs/api-reference/contribute.md (conflict)
src/docs/backend-smoke.md (conflict)
src/pkg/dashboard/contribute_landing.go (conflict)
src/pkg/dashboard/contribute_wall.go (conflict)
#18783 ✨ Evidence summary: change failures (reverts, reopened issues) per model same files dashboard/openapi.json
#18817 🐛 Customers: Inbox-style layout, less noise, no non-people suggested, attach to existing customer same files dashboard/openapi.json

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#187844e83e31` sha256:9ed17a5626dbe02e56097d871d9589a3ad7fe787e248c78dd0fde7222c6a4092 — download from the dashboard or `hivectl review evidence #18784`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant