Skip to content

feat(pipeline): let a stage declare it consumes no sibling's output - #60

Merged
pedromvgomes merged 1 commit into
mainfrom
feat/independent-stages
Aug 30, 2026
Merged

feat(pipeline): let a stage declare it consumes no sibling's output#60
pedromvgomes merged 1 commit into
mainfrom
feat/independent-stages

Conversation

@pedromvgomes

Copy link
Copy Markdown
Owner

Why

The stage graph is fixed on purpose, and stays fixed here. But dependency is a claim about the repository, and a repository can be right that it doesn't have one.

wardnet is the case. Its end2end suite rebuilds the daemon from source inside Docker, so it consumes nothing the build stage produces — its hand-rolled pr.yml ran the two concurrently (tests-e2e had needs: preflight only). gt's wiring gates end2end on build, so migrating it as-is serialises them:

today under gt
build stage (max leaf: Check Daemon 11m) parallel with e2e 11m
E2E Daemon 24m 24m
PR wall-clock ~24m ~35m

Eleven minutes added to every daemon PR to prove an edge that does not exist. Measured from run 32688637246, not estimated.

What

pipeline:
  ci:
    stages: [preflight, build, test, end2end]
    # end2end rebuilds the daemon from source and consumes nothing build produces.
    independent_stages: [end2end]

Renders:

  end2end:
    needs: [attest, preflight]
    if: needs.attest.outputs.validated != 'true' && needs.preflight.outputs.run-end2end != 'false'

What it deliberately cannot do

This is a statement about the repository, not a reordering knob. A stage named here drops its sibling dependencies and depends on preflight alone. It cannot add a dependency, invent an edge, or change the order stages render in. Removing an edge you don't have isn't rearranging the pipeline, so ## Shape's "a central pipeline every repo can rearrange is not a central pipeline" survives intact.

Two constraints fall out of that and are enforced rather than documented:

  • preflight is never dropped. It produces no artefact — it's the gate deciding whether a stage runs, and the rendered if: reads its outputs. Losing it from needs: would leave a condition that can never be true.
  • preflight cannot be declared independent. It has no sibling dependency to drop, so saying so is rejected rather than silently ignored. Same for a stage that isn't enabled, a duplicate, and any declaration while the pipeline is disabled.

The test I'd most want reviewed

TestIndependentStageIsStillGated. Detaching a stage must not drop it from ci-gate — a failing end2end still has to block the merge. Acquiring a silent hole in branch protection as a side effect of a performance tweak is the worst way this could go wrong, so it's pinned.

Also pinned: TestStagesKeepTheirDependenciesByDefault, so this can't quietly reshape the sixteen repos that don't use it.

Backwards compatibility

omitempty, and default rendering is untouched — verified by rendering every managed file on v1.5.0 (bc668c9) and on this branch and diffing: byte-identical, comments included.

Unlike #58 there's no fleet-wide comment churn, and gt's own repo stays compliant with no re-render (gt repo check → "All 8 managed file(s) match").

go test ./..., go vet ./..., gofmt clean.

Next

Needs shipping as v1.6.0 with v1 moved before wardnet's onboarding PR can land.

The stage graph is fixed on purpose, and stays fixed. But *dependency* is
a claim about the repository, and a repository can be right that it does
not have one.

    pipeline:
      ci:
        stages: [preflight, build, test, end2end]
        independent_stages: [end2end]

A stage named here drops its sibling dependencies and depends on
preflight alone. That is all it can do — it cannot add a dependency,
invent an edge, or change the order stages render in. Removing an edge
you do not have is not rearranging the pipeline.

preflight is never dropped, because it produces no artefact: it is the
gate deciding whether a stage runs at all, and the rendered `if:` reads
its outputs, so losing it from `needs:` would leave a condition that can
never be true. `preflight` itself cannot be declared independent, having
no sibling dependency to drop; saying so is rejected rather than ignored.

Detaching a stage does NOT remove it from ci-gate. A failing end2end
still blocks the merge; only its start time moves. There is a test
pinning that, because acquiring a silent hole in branch protection as a
side effect of a performance tweak is the worst way this could go wrong.

wardnet is the case it was added for. Its end2end suite rebuilds the
daemon from source inside Docker and consumes nothing the build stage
produces — its previous hand-rolled pipeline ran the two concurrently.
Serialising them added about eleven minutes to every daemon pull request,
24 to 35, to prove an edge that does not exist.

The field is omitempty and default rendering is untouched: rendering
every managed file on v1.5.0 and on this branch and diffing leaves them
byte-identical, comments included. Unlike #58 there is no fleet-wide
comment churn, and gt's own repo stays compliant without a re-render.
@github-actions

Copy link
Copy Markdown

bulwark

  • scan — no findings

  • coverage — go: 76.9% (baseline 76.8%); go patch: 93.6% (44/47 new lines; baseline 76.8%)

📦 Full bulwark output — complete scan and coverage logs.

@pedromvgomes
pedromvgomes merged commit afe13fd into main Aug 30, 2026
11 checks passed
@pedromvgomes
pedromvgomes deleted the feat/independent-stages branch August 30, 2026 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant