Skip to content

design(ci): require observed candidate-test VM validation before distributing workflow changes via v1 #1027

Closed

Description

@castrojo

What would you like to see?

Tests and reusable-workflow changes should be exercised against a real disposable image before they are distributed to consumer qualification gates. Lint, feature dry-run and helper unit tests do not prove GNOME/AT-SPI startup, host-vs-container probes, bootc reboots or result propagation work.

Verified starting point

Baseline main 6d36c72f52ec891ed66a8d947073854338581e38: .github/workflows/pr-validate.yml and unit-tests.yml trigger on pull_request/merge_group/main pushes but do not boot a candidate VM; e2e.yml is workflow_call only. update-v1-tag.yml advances managed v1 after main merges. The live main rules inspected during this investigation required Lint & syntax, Behave dry-run and pytest. That proves the observed required GHA checks, not the absence of all external lab signals.

PR validation workflow and managed tag publisher.

Scope / safety

Audit existing ghost-lab/commit-status and GHA runtime validation first; reuse an actually working signal instead of creating competing gates or inferring health from merged fixes. Execute the PR's real tests and workflow revision using correct test_repository/test_ref and immutable candidate identity, not defaults that silently select main/v1. Cover changed GUI/SSH/lifecycle paths where applicable. Untrusted PR code must not receive publishing credentials or execute through a privileged pull_request_target checkout. Separate read-only VM validation from trusted publication.

Acceptance

  • A reviewed change-path policy identifies the minimal representative VM jobs required for tests/shared, suite steps/hooks, runner, composite action and reusable-workflow changes; irrelevant docs-only changes need not boot.
  • Actual runs record the PR/merge revision and image digest and demonstrate that modified workflow/test code, not released main/v1 code, executed.
  • Setup failure, absent/zero required coverage or a deliberately broken assertion prevents qualification/distribution; an observed good candidate succeeds. Reuse the parent's fail-closed verdict contract.
  • Existing external statuses are audited end to end and reconciled with docs; no claim that ghost-lab works/is dead based only on prose or an action check-run API.
  • Demonstrate the path on a real PR and any required merge-group/publisher transition, with retained VM evidence. The outcome must be available before a failing test/workflow change reaches consumers.
  • Full affected workflow/helper test modules and actual VM paths pass; caller refs and matching docs/skills are updated consistently.

Human decision boundary

Required status checks, branch rules, v1 publication policy and CI credential/interface changes need explicit maintainer approval. The user requested tracking toward blocking tests, not unilateral ruleset/permission changes. Persistent lab orchestration stays in projectbluefin/lab. Parent: #1021.

Activity

  1. added theissue type on Oct 26, 2026
  2. added
    triage/acceptedScope accepted; ready for work. Use `/triage accepted`.
    and removed on Oct 27, 2026
  3. Danathar commented on Oct 28, 2026

    @Danathar
    Contributor

    Triage: accepted. Maintainer-authored design issue, part of the #1021 rollout plan.

  4. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    PR #1042 covers this issue: #1042

    The PR body says this is partial/non-closing work (Refs/Part of or leave-open wording), so Hive will not add a closing keyword. GitHub will not show its issue-list PR icon without a closing keyword; the stage comment is the link.

  5. added
    hive/partially-addressedA merged PR addressed only part of this issue; remainder still actionable
    and removed on Oct 28, 2026
  6. 3 remaining items

  7. Danathar commented on Oct 28, 2026

    @Danathar
    Contributor

    /hive decision Option A: a post-merge gate that holds v1. On a push to main, run scripts/vm_validation_plan.py on the merged diff, call ./.github/workflows/e2e.yml with test_ref set to the merged SHA and a digest-pinned image, and move v1 only when the #1022 fail-closed verdict (scripts/assert_passed.py) passes. Record the SHA and image digest in the job summary. Two conditions: (1) keep a manual workflow_dispatch path to move v1 anyway, for emergencies; (2) when a failed verdict holds v1 back, surface it visibly (a failed run plus an issue or a comment that names the held SHA), so a stuck tag is never silent. No skip-screenshot-publish input on e2e.yml for now; only Option B would need it, and #1037/#1050 already edit e2e.yml.

    Reason: consumers only use @v1, so gating the tag protects every image repo from a broken change, and only merged, trusted code runs on the VM, with no write tokens for PR code and no ruleset or permission changes. Option B would put a hand-approved, up-to-2-hour VM run in front of every fork PR, and most testsuite PRs come from first-time fork contributors. B can still be added later if A proves too loose.

  8. added
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Oct 28, 2026
  9. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    Approved by @Danathar. Decision: Option A: a post-merge gate that holds v1. On a push to main, run scripts/vm_validation_plan.py on the merged diff, call ./.github/workflows/e2e.yml with test_ref set to the merged SHA and a digest-pinned image, and move v1 only when the #1022 fail-closed verdict (scripts/assert_passed.py) passes. Record the SHA and image digest in the job summary. Two conditions: (1) keep a manual workflow_dispatch path to move v1 anyway, for emergencies; (2) when a failed verdict holds v1 back, surface it visibly (a failed run plus an issue or a comment that names the held SHA), so a stuck tag is never silent. No skip-screenshot-publish input on e2e.yml for now; only Option B would need it, and #1037/#1050 already edit e2e.yml.

    approved-direction is applied and Hive's own stage/045-needs-decision removed. Any hold label is left exactly as it was.

  10. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    Summary: I checked whether this is finished. It is not yet: the shared test tag that every image project uses is still moved forward on every change, with no test-machine run in front of it. The approved plan (option A) is still to be built, and the remaining change sits in a part of the repository that this agent is not allowed to change.


    Verified on main @ f077951d:

    All of the remaining work is under .github/workflows/. This lane's App token is minted without the Workflows permission, so I cannot push it. It needs a human or an ISSUES_PRS_MERGE agent to land. I have not opened a PR. I added hive/verified-open.

    — hive: agent=architect


    🐝 Hive Agent: architect | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: 9170788

  11. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    PR #1060 covers this issue: #1060

    The PR body says this is partial/non-closing work (Refs/Part of or leave-open wording), so Hive will not add a closing keyword. GitHub will not show its issue-list PR icon without a closing keyword; the stage comment is the link.

  12. hivecommons-hive commented on Oct 28, 2026

    @hivecommons-hive
    Contributor

    Fixed by #1055 (merged dadee6e). Closing. Reply /reopen if the problem persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions