Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 53 additions & 3 deletions .github/workflows/update-v1-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@
# image under test. When the candidate fails, the same suites boot again with
# the tests at the current v1 on the same digest (the baseline), and
# scripts/v1_gate_compare.py moves v1 only if every candidate failure also
# fails at v1. Anything it cannot explain holds v1.
# fails at v1. Anything it cannot explain holds v1. The failures it accepts
# come from the image or the VM, so scripts/v1_gate_preexisting.py files each
# one as an issue for the hive to quarantine or fix (#1143, #1153).
#
# The range starts at v1, not at the push's previous commit: if a newer push
# cancels a gated run, the next run still covers the cancelled change.
Expand Down Expand Up @@ -226,7 +228,9 @@ jobs:
with:
ref: ${{ needs.plan.outputs.sha }}
persist-credentials: false
sparse-checkout: scripts/v1_gate_compare.py
sparse-checkout: |
scripts/v1_gate_compare.py
scripts/v1_gate_preexisting.py
sparse-checkout-cone-mode: false

- name: Download candidate and baseline results
Expand All @@ -242,7 +246,7 @@ jobs:
run: |
set -euo pipefail
python3 scripts/v1_gate_compare.py artifacts "${IMAGE}" "${BASELINE_IMAGE}" \
--summary compare.md | tee -a "$GITHUB_OUTPUT"
--summary compare.md --preexisting preexisting.json | tee -a "$GITHUB_OUTPUT"
cat compare.md >> "$GITHUB_STEP_SUMMARY"

- name: Move or hold v1
Expand Down Expand Up @@ -342,3 +346,49 @@ jobs:
} >> "$GITHUB_STEP_SUMMARY"
echo "::error::v1 held at ${PREVIOUS:-none}: ${WHY}"
exit 1

# A failure that also fails at v1 comes from the image or the VM, not
# from the merged change, so it never holds v1. Left alone it keeps every
# later run red, the PR VM check included, until someone notices it by
# hand (#1143, #1153). File each one for the hive, once while an issue
# with the same title is open. Runs after a hold too: the failures it
# files are explained by v1 either way. Never fails the job.
- name: File pre-existing failures
if: always() && steps.compare.outcome == 'success'
continue-on-error: true
run: |
set -euo pipefail
[ -s preexisting.json ] || exit 0
python3 scripts/v1_gate_preexisting.py artifacts "${IMAGE}" preexisting.json "${RUN_URL}" > issues.jsonl
[ -s issues.jsonl ] || exit 0
OPEN_TITLES=$(gh issue list --state open --limit 1000 --json title --jq '.[].title')
: > filed.md
while IFS= read -r ISSUE; do
TITLE=$(jq -r .title <<<"${ISSUE}")
if grep -qxF -- "${TITLE}" <<<"${OPEN_TITLES}"; then
echo "Already open: ${TITLE}"
continue
fi
jq -r .body <<<"${ISSUE}" > body.md
URL=$(gh issue create --title "${TITLE}" --body-file body.md)
NUM=${URL##*/}
# One issue for many failures is a maintainer's call (likely the
# image as a whole); a single scenario is ready for the hive.
if [ "$(jq -r .needs_human <<<"${ISSUE}")" = "true" ]; then
LABELS=(kind/bug needs-human)
else
LABELS=(kind/bug triage/accepted)
fi
for LABEL in "${LABELS[@]}"; do
gh issue edit "${NUM}" --add-label "${LABEL}" > /dev/null \
|| echo "::warning::could not label #${NUM} ${LABEL}"
done
echo "- #${NUM} ${TITLE}" >> filed.md
done < issues.jsonl
if [ -s filed.md ]; then
{
echo "### Pre-existing failures filed for the hive"
echo
cat filed.md
} >> "$GITHUB_STEP_SUMMARY"
fi
14 changes: 14 additions & 0 deletions docs/skills/ci-ops/ops/references/candidate-vm-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,20 @@ and `.../check-runs`.
Also `priority/critical-urgent`;
- the issue has been open for more than 24 hours. Also
`priority/critical-urgent`.
4. Files the failures it let through. A scenario that fails at both the
merged SHA and `v1` comes from the image or the VM, so it never holds
`v1`, but it keeps every later run red (the PR VM check included) until
someone deals with it. The step "File pre-existing failures" runs
`scripts/v1_gate_preexisting.py` on the `--preexisting` output of
`v1_gate_compare.py` and opens one issue per scenario, titled
`e2e: <suite> fails on <image>, also at v1: <Feature> :: <Scenario>`,
labelled `kind/bug` and `triage/accepted`, for the hive to skip in VMs,
quarantine or fix. It files a title only when no open issue has it, so a
failure seen on every run is filed once. More than five at once becomes one
issue, `e2e: many scenarios fail on <image>, also at v1`, labelled
`needs-human`: that many points at the image as a whole. The step runs
after a hold too, and never fails the run. Before this, #1143 and #1153
were found by hand.

`lifecycle` is planned but not booted (`UNGATED_SUITES`): its upgrade
scenarios need a second, newer image to upgrade to (#706, #1023), and on one
Expand Down
11 changes: 10 additions & 1 deletion scripts/v1_gate_compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,17 @@
digest, so their suffixes differ.

Usage:
python3 scripts/v1_gate_compare.py ARTIFACT_DIR CANDIDATE_IMAGE BASELINE_IMAGE [--summary FILE]
python3 scripts/v1_gate_compare.py ARTIFACT_DIR CANDIDATE_IMAGE BASELINE_IMAGE [--summary FILE] [--preexisting FILE]

Prints ``verdict=move`` or ``verdict=hold`` and ``infrastructure=true`` or
``infrastructure=false`` for ``$GITHUB_OUTPUT``. ``infrastructure=true`` means
no failed suite produced test results on either side: the test machines most
likely never booted (a runner or install problem, as in #1123), so the hold is
not a test regression and needs a maintainer rather than a contributor.

``--preexisting FILE`` writes the failures that also fail at ``v1`` as JSON
(``{"suite": ["Feature :: Scenario", ...]}``) for
``scripts/v1_gate_preexisting.py``, which files them for the hive.
"""

from __future__ import annotations
Expand Down Expand Up @@ -164,10 +168,15 @@ def main(argv: list[str]) -> int:
parser.add_argument("candidate_image")
parser.add_argument("baseline_image")
parser.add_argument("--summary", type=Path)
parser.add_argument("--preexisting", type=Path)
args = parser.parse_args(argv[1:])
result = compare(args.artifacts, args.candidate_image, args.baseline_image)
if args.summary:
args.summary.write_text(summary_markdown(result), encoding="utf-8")
if args.preexisting:
args.preexisting.write_text(
json.dumps(result["preexisting"], indent=2) + "\n", encoding="utf-8"
)
print(f"verdict={result['verdict']}")
print(f"infrastructure={'true' if result['infrastructure'] else 'false'}")
return 0
Expand Down
211 changes: 211 additions & 0 deletions scripts/v1_gate_preexisting.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""Turn failures the v1 gate accepted as pre-existing into issues for the hive.

When a candidate failure also fails with the tests at ``v1`` on the same image
digest, ``scripts/v1_gate_compare.py`` lets ``v1`` move: the failure comes from
the image (or the VM), not from the merged change. Left alone, such a failure
keeps every later run red, including the PR VM check, until someone notices
it by hand (#1143, #1153). This script writes one issue per failing scenario
so the hive quarantines or fixes it without a person finding it first.

Usage:
python3 scripts/v1_gate_preexisting.py ARTIFACT_DIR CANDIDATE_IMAGE PREEXISTING_JSON RUN_URL

``PREEXISTING_JSON`` is the ``--preexisting`` file from ``v1_gate_compare.py``
(``{"suite": ["Feature :: Scenario", ...]}``). Prints one JSON object per line,
``{"title", "body", "needs_human"}``. The workflow files each title that has no
open issue yet, so a failure seen on every run is filed once.

More than ``MAX_ISSUES`` scenarios at once is treated as one image-wide
problem (a broken build, a missing package set) and becomes a single issue for
a maintainer, rather than a flood of per-scenario issues. Its title leaves
out the count, so a later run with a different count finds it still open.
"""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path

# v1_gate_compare.py is a sibling script, not a package. Add this directory so
# the import works both when run as ``python3 scripts/v1_gate_preexisting.py``
# and when imported as a module (pytest).
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from v1_gate_compare import FAILED_STATUSES, artifact_suffix # noqa: E402

MAX_ISSUES = 5
TITLE_LIMIT = 200


def image_name(image: str) -> str:
"""The image without its digest: ghcr.io/org/name:tag@sha256:... -> org/name:tag."""
name = image.split("@", 1)[0]
return name.split("/", 1)[1] if name.startswith("ghcr.io/") else name


def image_repo(image: str) -> str:
"""The GitHub repository that builds the image, for ghcr.io images."""
name = image.split("@", 1)[0]
if not name.startswith("ghcr.io/"):
return ""
path = name[len("ghcr.io/") :].rsplit(":", 1)[0]
return path if path.count("/") == 1 else ""


def repo_path(location: str) -> str:
"""behave's file:line, relative to the repo.

Some suites run from a checkout inside the guest and report it as a prefix
("bluefin-tests/tests/smoke/...:55"); others report "tests/...".
"""
if location.startswith("tests/"):
return location
at = location.find("/tests/")
return location[at + 1 :] if at >= 0 else location


def locations(artifacts: Path, image: str, suite: str) -> dict[str, str]:
"""Map "Feature :: Scenario" -> the scenario's file:line from behave's JSON."""
results = (
artifacts / f"e2e-results-{artifact_suffix(image)}-{suite}" / "results.json"
)
try:
features = json.loads(results.read_text(encoding="utf-8"))
except (OSError, ValueError):
return {}
found = {}
for feature in features if isinstance(features, list) else []:
for element in feature.get("elements", []):
if (
element.get("type") == "scenario"
and element.get("status") in FAILED_STATUSES
):
key = f"{feature.get('name', '?')} :: {element.get('name', '?')}"
found[key] = repo_path(element.get("location", ""))
return found


def title_for(image: str, suite: str, scenario: str) -> str:
title = f"e2e: {suite} fails on {image_name(image)}, also at v1: {scenario}"
return title if len(title) <= TITLE_LIMIT else title[: TITLE_LIMIT - 1] + "…"


def scenario_body(
image: str, suite: str, scenario: str, location: str, run_url: str
) -> str:
repo = image_repo(image)
where = (
f"`{location}`"
if location
else "the feature file (search for the scenario name)"
)
image_issue = (
f"an issue in `{repo}` (a maintainer files it if the hive cannot open issues there)"
if repo
else "an issue in the image's own repository (a maintainer files it)"
)
return f"""**Summary:** the scenario "{scenario}" in suite `{suite}` fails on `{image}` with the tests at the merged commit **and** with the tests at `v1`, so the post-merge `v1` gate accepted it as a pre-existing failure. Its cause is in the image or the VM, not in a testsuite change. Until it is handled, every run that boots `{suite}` stays red, including the PR VM check.

Filed automatically by the `v1` gate (`.github/workflows/update-v1-tag.yml`, step "File pre-existing failures").

## Evidence

- Scenario: {where}
- Image: `{image}`
- Run: {run_url} (job "Boot merged tests" and job "Boot v1 tests (baseline)", suite `{suite}`). The step summary lists it under "already present at v1"; the failing step and logs are in the `e2e-*` artifacts.

## Ask

1. Find out why it fails on this image: read the scenario's failing step and the journal, screenshot and coredump output in the run's artifacts.
2. Decide:
- **VM-only** (the VM lacks hardware or a service real machines have): skip it in VMs the way the suite already handles VM-only cases (for example `IGNORED_FAILED_UNITS_IN_VM` in `tests/smoke/features/steps/system_health_steps.py`, or an existing VM tag). Keep it on bare metal.
- **Image bug** (it would fail on real hardware too): tag the scenario `@quarantine` with a comment naming the cause, the way #1147 did, and say in the PR that a maintainer must file {image_issue}.
- **Test bug** (the test is wrong for this image): fix the test.
3. Don't raise a timeout to hide it, and don't delete the scenario.

## Acceptance criteria

- This scenario no longer fails in `{suite}` on the current `{image_name(image)}`: skipped in VMs, quarantined or fixed, with the reason next to the scenario. Other failures in the suite have their own issues.

## Test plan

- Unit tests and `behave --dry-run` (AGENTS.md gates).
- The PR VM check boots `{suite}` on the PR; link its run. If it does not run, a maintainer starts `manual.yml` with `suites={suite}` from the fix branch.

## Out of scope

- Changing the image itself, or `.github/workflows/e2e.yml`.
"""


def many_body(image: str, failures: list[tuple[str, str]], run_url: str) -> str:
listed = "\n".join(f"- `{suite}`: {scenario}" for suite, scenario in failures)
return f"""**Needs a maintainer:** {len(failures)} scenarios fail on `{image}` with the tests at the merged commit **and** at `v1`. That many at once points at the image as a whole (a broken or stale build, missing packages, a changed desktop), not at single tests, so the `v1` gate filed one issue instead of one per scenario.

Filed automatically by the `v1` gate (`.github/workflows/update-v1-tag.yml`, step "File pre-existing failures").

Run: {run_url}

{listed}

Decide whether the image or the tests need to change. Quarantining all of these would hide a broken image; once the cause is known, file per-scenario issues for what the hive should fix.
"""


def issues(artifacts: Path, image: str, preexisting: dict, run_url: str) -> list[dict]:
failures = [
(suite, scenario)
for suite in sorted(preexisting)
for scenario in sorted(preexisting[suite])
]
if not failures:
return []
if len(failures) > MAX_ISSUES:
title = f"e2e: many scenarios fail on {image_name(image)}, also at v1"
return [
{
"title": title,
"body": many_body(image, failures, run_url),
"needs_human": True,
}
]
found: dict[str, dict[str, str]] = {}
out = []
for suite, scenario in failures:
if suite not in found:
found[suite] = locations(artifacts, image, suite)
out.append(
{
"title": title_for(image, suite, scenario),
"body": scenario_body(
image, suite, scenario, found[suite].get(scenario, ""), run_url
),
"needs_human": False,
}
)
return out


def main(argv: list[str]) -> int:
if len(argv) != 5:
print(__doc__, file=sys.stderr)
return 2
artifacts, image, preexisting_file, run_url = (
Path(argv[1]),
argv[2],
Path(argv[3]),
argv[4],
)
try:
preexisting = json.loads(preexisting_file.read_text(encoding="utf-8"))
except (OSError, ValueError):
preexisting = {}
for issue in issues(artifacts, image, preexisting, run_url):
print(json.dumps(issue))
return 0


if __name__ == "__main__":
sys.exit(main(sys.argv))
1 change: 1 addition & 0 deletions scripts/vm_validation_plan.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@
("scripts/v1_gate.py", ()),
("scripts/v1_gate_compare.py", ()),
("scripts/v1_gate_pins.py", ()),
("scripts/v1_gate_preexisting.py", ()),
("scripts/vm_validation_plan.py", ()),
# Shared helpers: route by the path that imports them.
("tests/shared/kde_*", ("kde-smoke",)),
Expand Down
Loading
Loading