Repository navigation
ci(v1-gate): file failures the gate accepts as pre-existing for the hive - #1157
Conversation
When a scenario fails at both the merged SHA and v1 on the same image digest, the gate moves v1: the failure comes from the image or the VM. Nobody was told, so it kept every later run red until someone found it by hand (#1143, #1153). v1_gate_compare.py now writes those failures with --preexisting, and the new scripts/v1_gate_preexisting.py turns each into an issue (kind/bug, triage/accepted) with the scenario's file:line, the image and the run, asking the hive to skip it in VMs, quarantine it or fix the test. A new "File pre-existing failures" step files each title once while it is open, runs after a hold too, and never fails the run. More than five at once becomes a single needs-human issue, since that points at the image. Assisted-by: Claude Opus 5.5 via Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
49aa209
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @Danathar, head
Hive did not add the |
|
Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict approve, @Danathar was auto-promoted after 5 clean merged PRs in this hive's repositories. |
Summary: when the
v1gate accepts a failure because it also fails atv1, it now files an issue for the hive. These failures come from the image or the VM, not from a testsuite change, so they never holdv1. But nobody was told, so they kept every later run red until someone noticed by hand. #1143 and #1153 were both found that way. Once this lands, a new image bug reaches the hive on the first gate run that sees it.It also lets the PR VM check (#1142) be a plain red-means-broken check: image failures get quarantined or skipped by the hive instead of blocking every PR that boots a VM.
What I need from you: nothing. This isn't a human-merge path, so the hive merges it after review.
What changes
scripts/v1_gate_compare.py: new--preexisting FILEwrites the failures it explained byv1as JSON. Nothing else changes.scripts/v1_gate_preexisting.py(new): turns that JSON into one issue per scenario, with:file:line, taken from behave's JSON. Some suites report a guest checkout prefix such asbluefin-tests/tests/...; it's stripped;More than five at once becomes one issue titled
e2e: many scenarios fail on <image>, also at v1, labelledneeds-human: that many points at the image as a whole, not at single tests. The title has no count, so a later run with a different count still finds it open.update-v1-tag.yml: new step File pre-existing failures in the publish job:kind/bug+triage/accepted;v1either way;continue-on-error, so it never fails the run or blocks av1move.scripts/vm_validation_plan.py: the new script needs no VM.docs/skills/ci-ops/ops/references/candidate-vm-validation.md: step 4.Why not compare against
mainin the PR VM check insteadThat was the first plan, but the hive's merge gate blocks on any failed check, required or not (
FailOnAnyFailedCheckinsrc/pkg/github/merge_ci_gate.go). Eache2e.ymlVM job is its own check, so a comparison job at the end can't turn them green. A job-levelcontinue-on-errordoesn't help either: I checked on a throwaway branch, and the job's check run still concludesfailure. Keeping the PR check strict and keepingmainclean is the simpler fix.Tests
tests/unit/test_v1_gate_preexisting.py, 14 tests:bluefin-tests/prefix;compare --preexistingend to end;gh: it files and labels a new failure, skips a title that's already open, does nothing without failures, and sends many failures to a maintainer.gh) on the artifacts of gate run https://github.com/projectbluefin/testsuite/actions/runs/43035684640, which accepted smoke-b's Power panel failure at 17:58Z. Result: exactly one issue,e2e: smoke-b fails on projectbluefin/dakota:testing, also at v1: GNOME Settings smoke tests :: Power settings panel is accessible, pointing attests/smoke/features/gnome_settings.feature:55. A second run with that title open filed nothing.ruff check,actionlintand the gate's existing tests (test_v1_gate_compare.py,test_update_v1_tag_escalation.py,test_vm_validation_plan.py) pass.Note
#1153 already covers the Power panel failure under a hand-written title. Before this merges, #1153 gets retitled to the generated title above, so the first gate run finds it open instead of filing a duplicate.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn