Skip to content

ci(v1-gate): file failures the gate accepts as pre-existing for the hive - #1157

Merged
hivecommons-hive[bot] merged 1 commit into
mainfrom
ci/v1-gate-file-image-failures
Nov 11, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
mainfrom
ci/v1-gate-file-image-failures

Conversation

@Danathar

@Danathar Danathar commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary: when the v1 gate accepts a failure because it also fails at v1, it now files an issue for the hive. These failures come from the image or the VM, not from a testsuite change, so they never hold v1. But nobody was told, so they kept every later run red until someone noticed by hand. #1143 and #1153 were both found that way. Once this lands, a new image bug reaches the hive on the first gate run that sees it.

It also lets the PR VM check (#1142) be a plain red-means-broken check: image failures get quarantined or skipped by the hive instead of blocking every PR that boots a VM.

What I need from you: nothing. This isn't a human-merge path, so the hive merges it after review.

What changes

  • scripts/v1_gate_compare.py: new --preexisting FILE writes the failures it explained by v1 as JSON. Nothing else changes.

  • scripts/v1_gate_preexisting.py (new): turns that JSON into one issue per scenario, with:

    More than five at once becomes one issue titled e2e: many scenarios fail on <image>, also at v1, labelled needs-human: that many points at the image as a whole, not at single tests. The title has no count, so a later run with a different count still finds it open.

  • update-v1-tag.yml: new step File pre-existing failures in the publish job:

    • files each title only when no open issue has that title, so a failure seen on every run is filed once;
    • labels each issue kind/bug + triage/accepted;
    • runs after a hold too, because the failures it files are explained by v1 either way;
    • is continue-on-error, so it never fails the run or blocks a v1 move.
  • scripts/vm_validation_plan.py: the new script needs no VM.

  • docs/skills/ci-ops/ops/references/candidate-vm-validation.md: step 4.

Why not compare against main in the PR VM check instead

That was the first plan, but the hive's merge gate blocks on any failed check, required or not (FailOnAnyFailedCheck in src/pkg/github/merge_ci_gate.go). Each e2e.yml VM job is its own check, so a comparison job at the end can't turn them green. A job-level continue-on-error doesn't help either: I checked on a throwaway branch, and the job's check run still concludes failure. Keeping the PR check strict and keeping main clean is the simpler fix.

Tests

  • tests/unit/test_v1_gate_preexisting.py, 14 tests:
    • titles and locations, including the real bluefin-tests/ prefix;
    • the five-issue cap and the stable "many" title;
    • title truncation;
    • compare --preexisting end to end;
    • the workflow step itself, run with a stub gh: it files and labels a new failure, skips a title that's already open, does nothing without failures, and sends many failures to a maintainer.
  • Real data: I ran both scripts and the step (stub gh) on the artifacts of gate run https://github.com/projectbluefin/testsuite/actions/runs/43035684640, which accepted smoke-b's Power panel failure at 17:58Z. Result: exactly one issue, e2e: smoke-b fails on projectbluefin/dakota:testing, also at v1: GNOME Settings smoke tests :: Power settings panel is accessible, pointing at tests/smoke/features/gnome_settings.feature:55. A second run with that title open filed nothing.
  • ruff check, actionlint and the gate's existing tests (test_v1_gate_compare.py, test_update_v1_tag_escalation.py, test_vm_validation_plan.py) pass.

Note

#1153 already covers the Power panel failure under a hand-written title. Before this merges, #1153 gets retitled to the generated title above, so the first gate run finds it open instead of filing a duplicate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn

When a scenario fails at both the merged SHA and v1 on the same image
digest, the gate moves v1: the failure comes from the image or the VM.
Nobody was told, so it kept every later run red until someone found it
by hand (#1143, #1153).

v1_gate_compare.py now writes those failures with --preexisting, and the
new scripts/v1_gate_preexisting.py turns each into an issue (kind/bug,
triage/accepted) with the scenario's file:line, the image and the run,
asking the hive to skip it in VMs, quarantine it or fix the test. A new
"File pre-existing failures" step files each title once while it is
open, runs after a hold too, and never fails the run. More than five at
once becomes a single needs-human issue, since that points at the image.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.19820% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
scripts/v1_gate_preexisting.py 94.66% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 11, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
49aa209

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @Danathar, head 49aa209b9af7) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/update-v1-tag.yml
  • ci_gate_weakening — CI configuration is deleted, softened or has gate checks removed (Disables or softens CI gates (continue-on-error, || true, removed checks))
    • .github/workflows/update-v1-tag.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive
hivecommons-hive Bot merged commit 8a02572 into main Nov 11, 2026
7 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#115749aa209` sha256:66b1dfb69cf4ae8bb1fb8e314b762a7ca2226437fdaa6bfa1ee709a2d47974c5 — download from the dashboard or `hivectl review evidence #1157`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict approve, @Danathar was auto-promoted after 5 clean merged PRs in this hive's repositories.

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 11, 2026
Danathar added a commit that referenced this pull request Nov 11, 2026
Picks up #1158 (Power panel skip on hardware-free VMs) and #1157 so the
PR VM check's own run boots against today's main.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants