Skip to content

fix(e2e): stop common/smoke variant gates from producing false greens #1073

Closed

Description

@castrojo

Problem

A read-only coverage audit against shipping refs found that several runs can report success without exercising the image under test:

  • tests/common/features/environment.py selected @bluefin coverage only for image names containing bluefin or bazzite. Utah release gates and common PR e2e (ghcr.io/projectbluefin/common:e2e-pr-*) therefore skipped every common feature, although common PR e2e overlays common's shared and bluefin trees and Utah is Bluefin built on Fedora Hummingbird.
  • tests/smoke/features/environment.py read only IMAGE. With the default compose_image: true, IMAGE is ghcr.io/<owner>/testsuite-e2e:run-<id>, so smoke skipped @bluefin and @dakota_only scenarios. The GUI runner container also did not receive BASE_IMAGE.
  • .github/workflows/e2e.yml did not apply scripts/assert_passed.py to common, so an all-skipped common shard could pass.

Proposed fix

  • Use one shared image-family helper for common and smoke.
  • Read BASE_IMAGE before IMAGE, and pass BASE_IMAGE into the GUI runner container.
  • Treat bluefin, bazzite, utah, and common image names as Bluefin-family; keep Dakota separate.
  • Require at least one passing common scenario.
  • Update the affected skill guidance and add regression coverage.

Local evidence

  • ruff check tests/ --select E,F,W --ignore E501
  • python -m pytest tests/unit/ -q: 2,463 passed
  • python scripts/validate_docs.py
  • python scripts/generate_skill_index.py --check
  • behave --dry-run --no-summary tests/common/features

behave --dry-run tests/smoke/features fails locally on both main and the fix with the same 28 undefined qecore built-in steps; it needs the Fedora runner image.

Required VM validation

Run manual.yml from the branch with test_ref set to the branch head SHA and digest-pinned images for Dakota, Bluefin, Utah, and the relevant common candidate. Inspect scenario-level results for skip reasons, not just job conclusions.

Impact

The common-shard gate may correctly turn previously green runs red when a shard executes no passing scenario. This is intended: such runs do not establish coverage.

Assisted-by: GPT-6.1-Sol via omp

Activity

  1. self-assigned this
    on Oct 31, 2026
  2. castrojo commented on Oct 31, 2026

    @castrojo
    CollaboratorAuthor

    Coverage audit status

    Family-gate branch: fix/coverage-family-gates

    Runtime-tested at d2fbbf3d. HEAD d5953e8a adds documentation only. I compared each image against same-digest main baselines:

    Image Result against main
    Dakota +5 passing scenarios; failures 31 → 26; no new failures
    Bluefin Same 189 passing scenarios; failures 12 → 5; no new failures
    Utah +100 passing scenarios; 0 failures. This includes the common suite, which previously reported green after skipping everything.

    Utah software was cancelled by the existing Bazaar hang, which is fixed in #1075.

    This branch is not opened as a PR yet: it overlaps open #1037 in .github/workflows/e2e.yml and docs/skills/ci-ops/e2e-workflow/SKILL.md.

    Software and Compass branch: #1075

    VM-tested at 9f518f88:

    Image Result
    Dakota Passed
    Utah Passed
    Bluefin 21 passed, 2 failed, 12 skipped

    The two Bluefin failures are Bazaar Curated-tab failures. Bluefin pins an old curated config format that Bazaar 0.9.x rejects.

    Product findings

    • Bluefin Bazaar: the Curated tab is missing because of the incompatible curated config format.
    • Dakota: ships power-status-color@local from an old Bling pin, while the test expects power-status-color@projectbluefin.io.
    • Dakota: failures still exist on both main and this branch: Files/Settings AT-SPI failures, shell hook errors, and brew/distrobox availability failures.

    Coverage gaps still open

    Coverage is incomplete. The completeness review still lists missing tests in these areas:

    • common recipe and runtime behavior
    • Dakota-specific OS contracts
    • Utah flavor and startup contracts
    • Bluefin image contracts
    • Bazaar transactions
    • Compass runtime and UI behavior
    • Flatpak update and reinstall

    Some gaps need infrastructure that does not exist yet: hardware, an enforcing-SELinux lane, startup or installed-system qualification lanes, and suite selection in consumer workflows. Others are human decisions.

    Assisted-by: GPT-6.1-Sol via omp

  3. 35 remaining items

  4. hivecommons-hive commented on Nov 1, 2026

    @hivecommons-hive
    Contributor

    🛠️ Hive agent scanner (hive hosted-projectbluefin-knuckle-gjvq) started work on this issue at 2026-11-01T05:07:11Z. A pull request referencing this issue supersedes this mark; it lapses on its own at 2026-11-01T07:07:11Z if none appears.

  5. added
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    on Nov 1, 2026
  6. hivecommons-hive commented on Nov 1, 2026

    @hivecommons-hive
    Contributor

    Workflow half of #1073 (bot cannot push .github/workflows/*). Applies cleanly to current main; taken from castrojo's fix/coverage-family-gates branch.

    Changes: pass -e BASE_IMAGE and -D image=${BASE_IMAGE} into the GUI runner container (smoke family detection in the companion PR depends on it); add common to the pass-count gate; stop splitting common on Dakota; include utah in Flatpak cache conditions.

    Overlap: touches e2e.yml regions also edited by #1037; land after or rebase on it.

    Risk: credentials/owner-only infra

    diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
    index dae7a905..41452bf9 100644
    --- a/.github/workflows/e2e.yml
    +++ b/.github/workflows/e2e.yml
    @@ -110,20 +110,25 @@ jobs:
         steps:
           - name: Resolve suites
             id: resolve
    +        env:
    +          SUITES_INPUT: ${{ inputs.suites }}
    +          IMAGE_INPUT: ${{ inputs.image }}
             run: |
               JSON=$(python3 -c "
               import json, sys
               parts = [s.strip() for s in sys.argv[1].split(',') if s.strip()]
    +          image_name = sys.argv[2].lower().split('/')[-1].split(':')[0].split('@')[0]
               expanded = []
               for p in parts:
                   if p == 'smoke':
                       expanded.extend(['smoke-a', 'smoke-b', 'smoke-firefox'])
    -              elif p == 'common':
    +              elif p == 'common' and 'dakota' not in image_name:
                       expanded.extend([f'{p}-a', f'{p}-b'])
                   else:
    +                  # Dakota's common coverage is two Dakota-only features; a split leaves one all-skipped half.
                       expanded.append(p)
               print(json.dumps(expanded))
    -          " "${{ inputs.suites }}")
    +          " "$SUITES_INPUT" "$IMAGE_INPUT")
               echo "suites=${JSON}" >> "$GITHUB_OUTPUT"
               echo "Suites: ${JSON}"
     
    @@ -299,7 +304,7 @@ jobs:
           # suites must preload any explicitly-tested Flatpaks after SSH comes up.
           # Cache a small user Flatpak repo on the runner, then inject and deploy it in-VM.
           - name: Restore Flatpak download cache
    -        if: ${{ contains(inputs.image, '/bluefin') && steps.shard.outputs.suite_dir != 'common' && steps.shard.outputs.suite_dir != 'lifecycle' }}
    +        if: ${{ (contains(inputs.image, '/bluefin') || contains(inputs.image, '/utah')) && steps.shard.outputs.suite_dir != 'common' && steps.shard.outputs.suite_dir != 'lifecycle' }}
             id: flatpak-cache
             uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
             with:
    @@ -309,7 +314,7 @@ jobs:
                 flatpak-home-${{ runner.os }}-
     
           - name: Prime Flatpak download cache
    -        if: ${{ contains(inputs.image, '/bluefin') && steps.shard.outputs.suite_dir != 'common' && steps.shard.outputs.suite_dir != 'lifecycle' && steps.flatpak-cache.outputs.cache-hit != 'true' }}
    +        if: ${{ (contains(inputs.image, '/bluefin') || contains(inputs.image, '/utah')) && steps.shard.outputs.suite_dir != 'common' && steps.shard.outputs.suite_dir != 'lifecycle' && steps.flatpak-cache.outputs.cache-hit != 'true' }}
             run: |
               sudo apt-get update -q
               sudo apt-get install -y --no-install-recommends flatpak
    @@ -827,7 +832,7 @@ jobs:
               echo "${BOOT_TIME}" >> "$GITHUB_STEP_SUMMARY"
               echo '```' >> "$GITHUB_STEP_SUMMARY"
           - name: Install cached Flatpaks in VM
    -        if: ${{ contains(inputs.image, '/bluefin') && steps.shard.outputs.suite_dir != 'common' && steps.shard.outputs.suite_dir != 'lifecycle' }}
    +        if: ${{ (contains(inputs.image, '/bluefin') || contains(inputs.image, '/utah')) && steps.shard.outputs.suite_dir != 'common' && steps.shard.outputs.suite_dir != 'lifecycle' }}
             run: |
               SSH_COMMON="-i /tmp/vm_key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ControlMaster=auto -o ControlPersist=600 -o ControlPath=/tmp/ssh-ctrl-%C -o ConnectTimeout=3"
               CACHE_ROOT="${GITHUB_WORKSPACE}/.flatpak-cache-home/.local/share"
    @@ -1478,9 +1483,10 @@ jobs:
                     -e BEHAVE_RETRIES=${BEHAVE_RETRIES:-2} \
                     -e TARGET_IMAGE=\"${TARGET_IMAGE}\" \
                     -e IMAGE=\"${IMAGE}\" \
    +                -e BASE_IMAGE=\"${BASE_IMAGE}\" \
                     -e PYTHONPATH=/tmp/bluefin-tests \
                     ghcr.io/projectbluefin/testsuite:runner \
    -                \"python3 /tmp/bluefin-tests/tests/shared/behave_retry.py ${REMOTE_FEATURE_ARGS} --format json.pretty --outfile /tmp/results/results.json --no-capture ${BEHAVE_TAG_ARGS}\"
    +                \"python3 /tmp/bluefin-tests/tests/shared/behave_retry.py ${REMOTE_FEATURE_ARGS} -D image=${BASE_IMAGE} --format json.pretty --outfile /tmp/results/results.json --no-capture ${BEHAVE_TAG_ARGS}\"
                 " || BEHAVE_RC=$?
     
                 $SCP "bluefin-test@127.0.0.1:/tmp/results/." ./results/ 2>/dev/null || true
    @@ -1917,7 +1923,7 @@ jobs:
             if: steps.run.outputs.behave_rc != '0'
             run: exit "${{ steps.run.outputs.behave_rc }}"
     
    -      # Backstop: every desktop suite must have at least one genuinely passing
    +      # Backstop: every desktop suite and the common SSH suite must have at least one genuinely passing
           # scenario. behave exits 0 when a setup failure (failed_setup.txt) or a
           # self-disabled suite skips every scenario, so reading behave_rc alone is a
           # false green. assert_passed.py is the verdict contract for these suites (the
    @@ -1927,12 +1933,13 @@ jobs:
           # deliberately inapplicable shard (KDE webdriver stack could not be
           # installed) is passed through --skip-reason and is a passing terminal
           # state for the gate, not a product failure.
    -      - name: Assert desktop suite has passing scenarios
    +      - name: Assert suite has passing scenarios
             if: |
               (startsWith(env.SUITE_DIR, 'kde') ||
                env.SUITE_DIR == 'smoke' || env.SUITE_DIR == 'developer' ||
                env.SUITE_DIR == 'dx' || env.SUITE_DIR == 'software' ||
    -           env.SUITE_DIR == 'vanilla-gnome' || env.SUITE_DIR == 'bazzite') &&
    +           env.SUITE_DIR == 'vanilla-gnome' || env.SUITE_DIR == 'bazzite' ||
    +           env.SUITE_DIR == 'common') &&
               steps.run.outputs.behave_rc == '0'
             env:
               # Through env, not inline: the reason embeds strings read from the

    — hive: agent=scanner backend=copilot model=claude-opus-5.5 copilot=1.0.88

  7. Danathar commented on Nov 5, 2026

    @Danathar
    Contributor

    /hive decision Your token can push .github/workflows (the hive pushed e2e.yml in #1089, commit a017ce1). Open one PR with the workflow half described above (from castrojo's fix/coverage-family-gates branch); rebase it on current main and drop any part that has already landed.

  8. added
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Nov 5, 2026
  9. hivecommons-hive commented on Nov 5, 2026

    @hivecommons-hive
    Contributor

    Approved by @Danathar. Decision: Your token can push .github/workflows (the hive pushed e2e.yml in #1089, commit a017ce1). Open one PR with the workflow half described above (from castrojo's fix/coverage-family-gates branch); rebase it on current main and drop any part that has already landed.

    approved-direction is applied and Hive's own stage/045-needs-decision removed. Any hold label is left exactly as it was.

  10. hivecommons-hive commented on Nov 5, 2026

    @hivecommons-hive
    Contributor

    Fixed by #1083 (merged e06a534). Closing. Reply /reopen if the problem persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions