Skip to content

fix(ci): report nothing for a cancelled run instead of every package as failed - #271

Closed
clubanderson wants to merge 4 commits into
projectbluefin:mainfrom
clubanderson:fix/report-skips-cancelled-run
Closed

clubanderson wants to merge 4 commits into
projectbluefin:mainfrom
clubanderson:fix/report-skips-cancelled-run

Conversation

@clubanderson

@clubanderson clubanderson commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Why

Run https://github.com/projectbluefin/utah-packages/actions/runs/36205360031 was cancelled by hand (actor hanthor) one minute into rebuild0. No wave had uploaded an rpm-s<stage>-<package> artifact yet, and the report job ran on always(), so it read that silence as 215 failed packages and filed #267 naming the whole build list. Nothing actually failed: 39 jobs cancelled, 13 skipped, 0 failure.

What

  • .github/workflows/rebuild-rpms.yml — report runs on !cancelled() instead of always(), the same shape publish uses. A failed wave or a failed publish gate still reports and still drives the tracking issue; a cancelled run has no verdict and reports nothing.
  • tools/publish_gate.py — assert_gate_enforced now also checks the report job: rejects always(), requires !cancelled(), and requires it to wait for prepare, publish and every wave (precedence runs inside publish-repository.yml, so waiting on the final publish already covers it — the check deliberately does not list it).
  • tests/test_incremental_publish.py — updated the report-job test and added two negative cases (always() rejected; a report that does not wait for publish rejected).
  • Skill write-back (same PR, per AGENTS.md):
    • docs/skills/repeated-mistakes.md — entry 21: this is the second time the tracking issue spoke for a run that decided nothing (after the canary-on-main case, b858eee / fix(ci): keep a canary dispatched on main away from the tracking issue #261). Quick-check item added.
    • docs/skills/package-build-cache.md — no longer claims runs are "never cancelled"; a merge never cancels one, a human can, and says what the report does then.

Validation

  • just check — factory contract, validate, workflow quoting, runtime contract all pass.
  • pytest tests: 618 passed; the 1 failure (test_check_retry) is a pre-existing BSD-sed incompatibility on macOS, identical on main.
  • pre-commit run on the touched files — passes except a pre-existing actionlint SC2153 info in the untouched publish job.

The tracking issue #267 itself will be closed by the next healthy production run (the workflow already does that), or by this PR's Fixes #267 on merge.

Fixes #267

— hive: backend=copilot model=claude-fable-5.1

@castrojo

Copy link
Copy Markdown
Contributor

Verification & Status:

  • Diff: Modifies .github/workflows/rebuild-rpms.yml, tools/publish_gate.py, and tests/test_incremental_publish.py so the report job runs with !cancelled() rather than always(), preventing cancelled runs from registering all pending packages as failures on the factory tracking issue.
  • Merge status: CONFLICTING against current main. Specifically, commit 21e47c6 (#274 publish-per-wave) refactored .github/workflows/rebuild-rpms.yml and moved publication to publish-repository.yml, causing extensive merge conflicts in the workflow and gate logic.
  • Review state: 0 reviews recorded.

Next action: Author (@clubanderson) must rebase onto main and adapt the !cancelled() report logic to the refactored per-wave publish pipeline. Independent reviews (+2 requirement) required once rebased.

…as failed

Run 36205360031 was cancelled by hand one minute into rebuild0. No wave
had uploaded an rpm-s<stage>-<package> artifact yet, and the report job
ran on always(), so it read that silence as 215 failed packages and filed
the "Factory: packages failing on main" tracking issue naming the whole
build list (projectbluefin#267).

A cancelled run has no verdict. Gate report on !cancelled(), the same
shape publish already uses: a failed wave or a failed publish gate still
reports and still drives the tracking issue, a cancellation reports
nothing. assert_gate_enforced now rejects always() on the report job and
requires it to wait for prepare, precedence, publish and every wave, and
the workflow test covers both.

Skill write-back: repeated-mistakes gains entry 21 (this is the second
time the tracking issue spoke for a run that decided nothing, after the
canary-on-main case in b858eee), and package-build-cache no longer claims
runs are never cancelled -- a merge never cancels one, a human can.

Fixes projectbluefin#267

Hive-Run: projectbluefin#267
Hive-Plan: factory-report-cancelled-run
Hive-Spec: factory-report#cancelled-run-no-verdict
Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the fix/report-skips-cancelled-run branch from fa55624 to e74a1a8 Compare September 26, 2026 05:28
@clubanderson

Copy link
Copy Markdown
Contributor Author

Rebased onto main (now on top of #277, past the #274 per-wave publish refactor) and adapted to the new layout:

  • report.if is !cancelled() on main's job shape; report.needs follows main and no longer lists precedence, which now runs inside publish-repository.yml and is covered by waiting on the final publish.
  • _assert_report_has_a_verdict is called from assert_gate_enforced on rebuild-rpms.yml (not from _assert_publication, which only sees the reusable workflow), and its required needs are prepare, publish and every wave.

python3 -m unittest discover -s tests passes for the publish-gate suite and assert_gate_enforced(load_workflow(), load_workflow(PUBLISH_WORKFLOW)) is green on this head. Conflicts are cleared; ready for review.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. Verified on head e74a1a8:

  • .github/workflows/rebuild-rpms.yml:931-936 — the publish job already gates on !cancelled() && needs.prepare.result == 'success' && ..., so the new report gate at :961 is the same shape as the job it depends on, as the comment claims. The failed-wave path is preserved: a wave that fails leaves publish skipped, not cancelled, and !cancelled() is true for skipped upstream jobs, so the report still runs and still reads the artifact list.
  • tools/publish_gate.py:_assert_report_has_a_verdict rejects always(), requires !cancelled(), and requires needs to include prepare, publish, and every stage — and it is wired into assert_gate_enforced (:416), so the CLI no-arg mode enforces it too, not just pytest.
  • python3 -m pytest tests/ on this head: 655 passed (the two new negative tests included).

Docs in package-build-cache.md and repeated-mistakes.md §21 match the behaviour. Nothing else in the diff.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at e74a1a8. Gates the rebuild report job on !cancelled() instead of always(), preventing cancelled runs from reporting all unbuilt packages as failed in the tracking issue. Tests in tests/test_incremental_publish.py pass and assert_gate_enforced verifies the rule. Awaiting second independent review and Canary run on a repo-owned branch to hit merge threshold.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency

  • (low) PR body says the new gate "requires it to wait for prepare, precedence, publish and every wave", but tools/publish_gate.py:467 checks only ("prepare", "publish", *STAGES) and the comment at :465-466 deliberately excludes precedence (it runs inside publish-repository.yml). Code is right; the PR description overstates it — worth correcting the body so the next reader doesn't hunt for a missing check.

No findings from: correctness, security, intent-alignment, style.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson

Copy link
Copy Markdown
Contributor Author

Thanks — fixed the PR body: the assert_gate_enforced bullet now says the report gate waits for prepare, publish and every wave, and notes that precedence is intentionally not listed because it runs inside publish-repository.yml (so waiting on the final publish covers it), matching tools/publish_gate.py:465-467. No code change.

@castrojo
castrojo requested a review from Danathar September 27, 2026 00:34

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • (low, in-scope) The report if: change (.github/workflows/rebuild-rpms.yml:969) has no run proving it: the Canary check on this PR exits in 3s with "a fork pull request cannot publish the canary image; a maintainer must push this branch to the repository" (run 36324155550). Your own quick-check (docs/skills/repeated-mistakes.md:497 "Is there a run, on this head, that reached the gate") is unmet. Low risk — the same !cancelled() shape already gates rebuild1..13 and publish* at rebuild-rpms.yml:587–909 — but a maintainer push of the branch would close the gap.

docs-currency

  • (info, in-scope) PR body says the write-back is "entry 21"; the diff adds ## 22 (docs/skills/repeated-mistakes.md:464, following existing ## 21 at :445). The file is right; the body is off by one.

No findings from: security, intent-alignment, style.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff at 11380ea against main. The report job now runs on !cancelled() instead of always(), the same shape as publish, so a failed wave or failed publish gate still reports and a cancelled run does not. tools/publish_gate.py rejects always() on report and requires it to wait for prepare, publish and every wave.

Ran on 11380ea: python3 -m unittest discover -s tests (694 tests, OK, 1 skipped) and python3 tools/publish_gate.py (exit 0). Also merged onto current main 896ad14 (clean) and reran both: OK. Put always() back on the report job in a scratch copy and assert_gate_enforced rejects it.

CI on 11380ea: Tests run 36324155138 passed (unit tests, factory config, onboarding, runtime contract, quoting). Canary in run 36324155550 failed after 3s with "a fork pull request cannot publish the canary image; a maintainer must push this branch to the repository". That is the fork guard, not the change, but it means no canary has run on this head.

@castrojo

Copy link
Copy Markdown
Contributor

Carried unchanged as #300 from an in-repo branch so the Canary can run past the fork guard (the only failing required check here). I'll merge #300 once its Canary is green and close this one then; nothing needed from you.

@hanthor hanthor closed this in #300 Sep 29, 2026
hanthor added a commit that referenced this pull request Sep 29, 2026
…ncelled-run

fix(ci): report nothing for a cancelled run instead of every package as failed (carries #271)
@hanthor

hanthor commented Sep 29, 2026

Copy link
Copy Markdown
Member

Shepherding note: the carry PR #300 has merged (or is in flight for #233). This original can be closed as superseded once you've confirmed the carry covers your change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Factory: packages failing on main

4 participants