Skip to content

ci: reclaim runner disk before image builds - #929

Closed
castrojo wants to merge 3 commits into
projectbluefin:mainfrom
castrojo:fix/issue-796-enospc
Closed

castrojo wants to merge 3 commits into
projectbluefin:mainfrom
castrojo:fix/issue-796-enospc

Conversation

@castrojo

@castrojo castrojo commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Reclaim hosted-runner disk space immediately after checkout in the common image build workflow.
  • Use the org-standard ublue-os/remove-unwanted-software@v10 action with extra-squeeze: "true".
  • Document the disk-space preparation pattern and source-verification commands in docs/skills/ci.md.

Closes #796

Evidence

Issue #796 identifies intermittent ENOSPC while large OSTree/OCI payloads are extracted on GitHub-hosted runners. The shared projectbluefin/actions/bootc-build/setup-runner action already uses this exact cleanup action and pin. The new step applies the same source-backed mitigation before this repository's image build.

Validation

  • git diff --check — passed.
  • Workflow assertions — passed: cleanup step is after checkout and before tag generation, uses the verified v10 pin, and sets extra-squeeze: "true".
  • just check — not run; just is unavailable in this environment.
  • pre-commit run --all-files — not run; pre-commit is unavailable in this environment.
  • Full image build — not run; this is a workflow-only change.

Free hosted-runner space before extracting image layers, and document the shared disk-space preparation pattern.

Assisted-by: goose
@castrojo
castrojo requested review from a team and repires as code owners August 4, 2026 19:23
@bketelsen

Copy link
Copy Markdown
Collaborator

I audited the actual ISO workflows for #796. This workflow already calls projectbluefin/actions/bootc-build/setup-runner, whose current implementation runs remove-unwanted-software v10 with extra-squeeze, so adding the action here duplicates cleanup in the regular Common image build and does not cover ISO extraction.

The remaining uncovered path is projectbluefin/iso’s reusable Titanoboa workflow; I opened projectbluefin/iso#69 with the cleanup immediately before the ISO job’s dependency installation and checkout.

Separately, this PR’s validation failure is from docs/skills/ci.md: it has no required front matter and is missing from docs/SKILL.md and docs/skills/index.json.

castrojo added a commit that referenced this pull request Aug 7, 2026
## Problem

The `validate` check has been producing false-negative failures across
the open PR backlog. At time of writing, 9 of 19 open PRs (#893, #917,
#924, #926, #929, #931, #932, #933, #934) showed a red `validate` check
unrelated to their own content:

```
error: docs/skills/index.json is stale. Run scripts/generate_skill_index.py --write
```

## Root cause

`scripts/generate_skill_index.py` stamped `generated_at` with
`date.today()` at **script-execution time**, then byte-compared the
result against the committed `docs/skills/index.json`.

That means the check only passed if CI happened to run on the *same
calendar day* the index was last regenerated on `main` — entirely
independent of whether any skill doc actually changed. Any PR whose CI
ran a day or more after the last regeneration failed.

Reproduced locally with zero content edits: `--check` failed purely
because days had passed since the last commit to `index.json`.

This matters because `validate` is a **required status check** in the
merge queue ruleset, so the noise trains reviewers to ignore a gate that
is supposed to block merges.

## Fix

Only advance `generated_at` when the catalog's actual content
(`schema_version` + `skills`) differs from what is committed. `--write`
becomes idempotent when nothing changed; `--check` tolerates a stale
date as long as the skill data matches.

## Verification

Proved in both directions, with a regression test for each:

- **No false negative** — backdated the committed index to `2020-01-01`
with zero skill content changes; `--check` now passes (previously
failed). Covered by
`test_generate_skill_index_check_tolerates_stale_generated_at`.
- **Real drift still caught** — edited a skill's front matter without
regenerating; `--check` still fails with `index.json is stale`. Covered
by `test_generate_skill_index_check_still_fails_on_real_drift`.

`just check` passes. Skill doc test suite: 8 passed.

## Alternatives rejected

- *Compare against the merge result instead of PR head* — doesn't
address the cause; the merge tree still gets stamped "today".
- *Scope the check to changed files* — would blind it to a PR that edits
a skill doc without regenerating the index.
- *Stop committing generated files* — far larger blast radius for a
date-comparison bug.
- *Auto-commit bot step in CI* — added complexity over a small fix.

Assisted-by: Claude Sonnet 5 via GitHub Copilot

Co-authored-by: Jorge Castro <jorge@projectbluefin.io>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@castrojo

castrojo commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Closing as already covered upstream — the disk reclaim this adds is a duplicate of one that already runs.

The Setup runner step immediately above the insertion point resolves to projectbluefin/actions/bootc-build/setup-runner, which already runs the identical action, at the identical SHA, with the identical input:

# projectbluefin/actions bootc-build/setup-runner/action.yml:41
- name: Maximize build space (remove-software)
  uses: ublue-os/remove-unwanted-software@695eb75bc387dbcd9685a8e72d23439d8686cba6 # v10
  with:
    extra-squeeze: "true"

Re-derive with:

gh api repos/projectbluefin/actions/contents/bootc-build/setup-runner/action.yml \
  --jq .content | base64 -d | grep -n -A5 -B3 'Maximize build space'

So the space is already reclaimed before the added step runs; the second invocation would find little to remove while still costing runner time on every build, across both arches.

Two smaller notes for anyone revisiting this:

  • The added docs/skills/ci.md says "Keep this step immediately after checkout", but the diff places it after Setup runner. If this ever is re-added, doc and diff should agree.
  • ci: ISO builds fail intermittently due to ENOSPC during OSTree extraction #796 was specifically about ISO builds (build-iso-bluefin.yml, non-composefs OSTree variants unpacking ~120 layers). common has no ISO workflows, so build.yml was not the workflow experiencing ENOSPC.

The SHA pin itself was correct and correctly commented — the instinct was right, it just already happens via the shared action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: ISO builds fail intermittently due to ENOSPC during OSTree extraction

2 participants