Skip to content

OSAC-3353: GHA workflow to build and validate base SNO snapshot - #34

Closed
omer-vishlitzky wants to merge 4 commits into
osac-project:mainfrom
omer-vishlitzky:feat/OSAC-3353-base-sno-gha
Closed

OSAC-3353: GHA workflow to build and validate base SNO snapshot#34
omer-vishlitzky wants to merge 4 commits into
osac-project:mainfrom
omer-vishlitzky:feat/OSAC-3353-base-sno-gha

Conversation

@omer-vishlitzky

@omer-vishlitzky omer-vishlitzky commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a GitHub Actions workflow (build-base-sno.yaml) that automates the full lifecycle of a base SNO snapshot:

  • build-snapshot: Creates a snapshot from a running source cluster using scripts/snapshot_base.py
  • validate-snapshot (OSAC-3357): Boots a fresh clone from the snapshot and validates node readiness, ClusterOperator health, and cluster connectivity via cluster-tool verify
  • publish-snapshot: Pushes the validated snapshot to the OCI registry (Quay)

Workflow inputs

Input Description
ocp_version OCP version slug (e.g. 4-22)
source Clone ID of running source cluster (8 hex chars from virsh list)

Prerequisites

  • A running SNO cluster on the baremetal runner (source clone ID required)
  • The scripts/ directory with snapshot_base.py (OSAC-3352)

Secrets and variables

Name Type Default
PULL_SECRET secret (required)
SERVER var rdu07
DATA_PATH var /data/cluster-tool
QUAY_REGISTRY var quay.io/osac-project/cluster-flavors

Depends on OSAC-3352 for the scripts/ directory (snapshot_base.py).

Test plan

  • Verify workflow YAML is valid (actionlint or manual review)
  • Run workflow via workflow_dispatch on a baremetal runner with a running source cluster
  • Confirm snapshot is created, validation passes, and image is pushed to Quay
  • Pull the pushed image on a different machine and boot to confirm end-to-end

Summary by CodeRabbit

  • New Features
    • Added an automated pipeline for building and publishing base single-node OpenShift snapshots.
    • Added snapshot validation that checks node readiness, cluster operator health, and connectivity before publication.
    • Added support for manually triggered runs with a specified OpenShift version and source snapshot.
    • Added automatic cleanup of temporary validation resources and credentials.

…snapshots

Adds a GitHub Actions workflow (build-base-sno.yaml) that automates the
full lifecycle of a base SNO snapshot: create from a running source
cluster, validate by booting a clone and checking node/operator health,
then push to the OCI registry.

The workflow runs on self-hosted baremetal runners and accepts the OCP
version and source clone ID as workflow_dispatch inputs. Validation
(OSAC-3357) checks node readiness, ClusterOperator health, and runs
cluster-tool verify for DNS and API connectivity.

Depends on OSAC-3352 for the scripts/ directory (snapshot_base.py).

Signed-off-by: Omer Vishlitzky <ovishlitzky@gmail.com>
Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: omer-vishlitzky <omer.vishlitzky@gmail.com>
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@omer-vishlitzky, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1d928d2c-746c-4c8a-8e7a-d24baa44924d

📥 Commits

Reviewing files that changed from the base of the PR and between 17f285d and 82a7ca2.

📒 Files selected for processing (1)
  • .github/workflows/build-base-sno.yaml

Walkthrough

The new workflow builds a base SNO snapshot from a source cluster, boots and validates a fresh cluster, verifies connectivity, destroys the validation cluster, and publishes the validated snapshot to a registry.

Changes

Base SNO snapshot workflow

Layer / File(s) Summary
Build, validate, and publish snapshot
.github/workflows/build-base-sno.yaml
Adds manually dispatched and main-push triggers, builds snapshots with cluster-tool, validates node and ClusterOperator health on a fresh cluster, cleans up resources and secrets, and publishes successful snapshots.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitHubActions
  participant SourceCluster
  participant ValidationCluster
  participant SnapshotRegistry
  GitHubActions->>SourceCluster: build snapshot from SOURCE
  GitHubActions->>ValidationCluster: boot snapshot and check node/operator health
  GitHubActions->>ValidationCluster: verify connectivity and destroy cluster
  GitHubActions->>SnapshotRegistry: publish validated snapshot
Loading

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (2 errors)

Check name Status Explanation Resolution
No-Injection-Vectors ❌ Error workflow_dispatch.ocp_version becomes FLAVOR_NAME and is passed to cluster-tool push; that code runs subprocess.run(..., shell=True) with unquoted args.name interpolation. Whitelist the workflow input and refactor cluster-tool to pass argv lists or shlex.quote every interpolated value before shell execution.
No-Sensitive-Data-In-Logs ❌ Error The workflow runs oc get nodes, which prints node names and can expose internal hostnames in CI logs. Replace the node list dump with a non-verbose readiness check (e.g. oc wait or a name-free status probe) and avoid echoing any secret values.
✅ Passed checks (9 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the new GitHub Actions workflow and its main build/validation purpose.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No-Hardcoded-Secrets ✅ Passed No hardcoded secrets found; the workflow only references secrets.PULL_SECRET and uses non-sensitive paths/defaults.
No-Weak-Crypto ✅ Passed The only changed file is a workflow YAML, and it contains no weak-crypto APIs, custom crypto, or secret/token comparisons.
Container-Privileges ✅ Passed PASS: The only changed file is a GitHub Actions workflow, and it contains no privileged/container securityContext settings.
Ai-Attribution ✅ Passed HEAD commit has an Assisted-by trailer for Claude Code, and no Co-Authored-By AI attribution was found.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/build-base-sno.yaml:
- Line 40: Add SAST and SCA dependency-scanning steps to the workflow before the
cluster operation steps, ensuring both run before the production snapshot build
or publish proceeds. Use the existing workflow conventions and security tooling
configuration where available.
- Around line 150-153: Remove the `|| true` suppression from the “Destroy
validation cluster” step so failures from `cluster-tool destroy validate` fail
the job. If needed, add only bounded retries while preserving the step’s
`always()` execution.
- Around line 23-25: Remove the no-op push trigger from the workflow’s on
configuration, specifically the push block matching the main branch and
scripts/** paths. Preserve the workflow_dispatch trigger and existing job
behavior.
- Around line 60-65: The pull secret is created only in build-snapshot and can
be unavailable to validate-snapshot on another runner, while cleanup may be
skipped on failure. Update the build-snapshot, validate-snapshot, and
publish-only job steps to create the secret within each job that consumes it,
remove the cross-job /tmp dependency, and add cleanup steps guarded with if:
always() so the secret is removed after success or failure.
- Line 35: Declare the custom baremetal runner label in the repository’s
actionlint configuration, preserving the existing self-hosted runner mapping
used by the runs-on entries in build-base-sno.yaml. Add only the necessary
runner-label configuration and keep the workflow unchanged.
- Around line 159-174: Update the publish-snapshot job and cluster-tool push
flow to expose the pushed manifest digest, then sign the immutable image
reference using cosign keyless signing as registry@digest. Grant id-token: write
only in this job’s permissions, while preserving existing read access and
leaving build/validate jobs unchanged; ensure the digest output from
cluster-tool push is consumed by the signing step.
- Around line 67-72: Update the “Create base snapshot” step to construct
KUBECONFIG inside its run script using the shell’s $HOME value and
inputs.source, instead of interpolating env.HOME in the workflow expression.
Preserve the existing kubeconfig path structure and SOURCE environment variable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8d43fe13-0fe6-4932-a257-0d154a69af25

📥 Commits

Reviewing files that changed from the base of the PR and between ade62ac and 17f285d.

📒 Files selected for processing (1)
  • .github/workflows/build-base-sno.yaml

Comment thread .github/workflows/build-base-sno.yaml Outdated
Comment thread .github/workflows/build-base-sno.yaml Outdated

jobs:
build-snapshot:
runs-on: [self-hosted, baremetal]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

fd -HI '^actionlint\.ya?ml$' . .github 2>/dev/null || true
rg -n -C2 'self-hosted-runner:|labels:|baremetal' .github actionlint.yaml 2>/dev/null || true

Repository: osac-project/cluster-tool

Length of output: 1531


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== workflow outline =="
sed -n '1,190p' .github/workflows/build-base-sno.yaml

echo "== actionlint config files =="
fd -HI 'actionlint\.ya?ml$|^actionlint$|\.actionlint$' . .github 2>/dev/null || true

echo "== actionlint references in repo =="
rg -n 'actionlint|self-hosted-runner|self-hosted runners|baremetal|github.com/rhysd/actionlint' .github . 2>/dev/null | head -200

Repository: osac-project/cluster-tool

Length of output: 7755


Declare the custom baremetal runner label to actionlint.

baremetal is used as a self-hosted runner label in .github/workflows/build-base-sno.yaml at lines 35, 76, and 157, but the repository has no actionlint configuration. Add it to the repo’s actionlint config while preserving the intended self-hosted runner mapping.

🧰 Tools
🪛 actionlint (1.7.12)

[error] 35-35: label "baremetal" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file

(runner-label)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-base-sno.yaml at line 35, Declare the custom
baremetal runner label in the repository’s actionlint configuration, preserving
the existing self-hosted runner mapping used by the runs-on entries in
build-base-sno.yaml. Add only the necessary runner-label configuration and keep
the workflow unchanged.

Source: Linters/SAST tools

github.event_name == 'workflow_dispatch'
permissions:
contents: read
steps:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Add SAST and SCA before operating the cluster.

This workflow has no SAST or dependency-scanning stage despite building and publishing a production snapshot. As per path instructions, SAST/SCA steps in pipeline.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-base-sno.yaml at line 40, Add SAST and SCA
dependency-scanning steps to the workflow before the cluster operation steps,
ensuring both run before the production snapshot build or publish proceeds. Use
the existing workflow conventions and security tooling configuration where
available.

Source: Path instructions

Comment thread .github/workflows/build-base-sno.yaml
Comment thread .github/workflows/build-base-sno.yaml Outdated
Comment thread .github/workflows/build-base-sno.yaml Outdated
Comment on lines +159 to +174
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false

- name: Push snapshot to registry
run: |
./cluster-tool push "$FLAVOR_NAME" \
--registry "$REGISTRY" \
--tag "$FLAVOR_NAME"
env:
FLAVOR_NAME: sno-${{ inputs.ocp_version }}
REGISTRY: ${{ vars.QUAY_REGISTRY || 'quay.io/osac-project/cluster-flavors' }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Tracked workflow files:"
git ls-files '.github/workflows/**/*' | sed -n '1,200p'

echo
echo "Target workflow outline/size:"
wc -l .github/workflows/build-base-sno.yaml
sed -n '1,230p' .github/workflows/build-base-sno.yaml

echo
echo "Cosign/sigstore references:"
rg -n --hidden --glob '!node_modules/**' --glob '!vendor/**' 'cosign|sigstore|oidc|permissions:|id-token|registry@digest|push snapshot|Push snapshot|cluster-tool push' .github . || true

Repository: osac-project/cluster-tool

Length of output: 7144


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Repository root files:"
git ls-files | rg '(^|/)(main|init|go|pyproject|README|Makefile|Dockerfile|cluster-tool|push|registry|oidc|cosign)' || true

echo
echo "Search push command implementation references:"
rg -n --hidden --glob '!node_modules/**' --glob '!vendor/**' --glob '!dist/**' --glob '!build/**' 'func .*push|class .*Push|def .*push|push|registry|oci|cosign|sigstore|oidc|digest|image|artifact' --glob '!*.md' | sed -n '1,240p'

echo
echo "Candidate source files:"
git ls-files | sed -n '1,240p'

Repository: osac-project/cluster-tool

Length of output: 257


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "cluster-tool type and size:"
file cluster-tool
wc -c cluster-tool
head -c 8000 cluster-tool | sed -n '1,240p'

echo
echo "Read-only token usage and registry signature references in cluster-tool:"
python3 - <<'PY'
import pathlib, re
p=pathlib.Path('cluster-tool')
text=p.read_text(errors='replace')
pat=r'\b(permissions|id-token|cosign|sigstore|oidc|registry@digest|digest|push\s*$|cmd\.push|push_cmd|oci|image)\b|^\s*echo\s+".*digest|PRINT|print\('
for i,line in enumerate(text.splitlines(),1):
    if re.search(pat,line):
        print(f"{i}: {line}")
PY

echo
echo "Search cosign/sigstore usage in repo:"
rg -n --hidden --glob '!node_modules/**' --glob '!vendor/**' 'cosign|sigstore|oidc|registry@digest|sign \(|sign.*digest|cosign sign|cluster-tool sign' . || true

Repository: osac-project/cluster-tool

Length of output: 233


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "cluster-tool first lines and size:"
wc -c cluster-tool
sed -n '1,240p' cluster-tool

echo
echo "Push-related text in cluster-tool:"
rg -n --hidden --glob '!node_modules/**' --glob '!vendor/**' 'push\s*$|cmd(.*)push|function .*push|def .*push|subprocess\(|cosign|sigstore|oidc|id-token|token|digest|image|OCI|registry|quay|docker|podman|skopeo' cluster-tool README.md .claude || true

echo
echo "Repo-wide cosign/sigstore/identity token/digest/sign references:"
rg -n --hidden --glob '!node_modules/**' --glob '!vendor/**' 'cosign|sigstore|oidc|id-token|registry@digest|digest|sign \(|cosign sign|cosign sign-blob|cluster-tool sign|skopeo|podman sign' . || true

Repository: osac-project/cluster-tool

Length of output: 22093


Sign the published snapshot by immutable digest.

publish-snapshot only tags and pushes registry:tag; no Sigstore/cosign signing is performed. Have cluster-tool push expose the pushed manifest digest, then sign registry@digest with a keyless identity. Add id-token: write only to this job so GitHub’s OIDC signing flow can request the signing identity token while keeping signing out of the build/validate jobs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-base-sno.yaml around lines 159 - 174, Update the
publish-snapshot job and cluster-tool push flow to expose the pushed manifest
digest, then sign the immutable image reference using cosign keyless signing as
registry@digest. Grant id-token: write only in this job’s permissions, while
preserving existing read access and leaving build/validate jobs unchanged;
ensure the digest output from cluster-tool push is consumed by the signing step.

Source: Path instructions

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

Code Review Findings

CRITICAL — Snapshot is pushed to registry BEFORE validation runs

build-base-sno.yaml:70snapshot_base.py pushes the snapshot to the OCI registry at step 6/6 (cluster_tool('push', ...)) during the build-snapshot job, before the validate-snapshot job ever runs. The publish-snapshot job then pushes the same image again redundantly. This defeats the entire validate-then-publish pipeline design — a bad snapshot reaches the registry before validation.

Fix: Either (a) add a --skip-push flag to snapshot_base.py so the workflow's publish-snapshot job is the sole publisher, or (b) split snapshot_base.py into separate snapshot and push steps.

MAJOR — Pull secret cross-job filesystem dependency

build-base-sno.yaml:97 — The validate-snapshot job reads /tmp/pull-secret.json written by the build-snapshot job. These are separate GitHub Actions jobs. Even on self-hosted runners, GHA does not guarantee they land on the same runner. OSAC runs multiple runners per baremetal machine (runner-01, runner-02, etc. via action-runners-setup.sh), so a different runner could pick up the next job.

Fix: Write the pull secret in each job that needs it, or consolidate into a single job.

MAJOR — Credential leak on validation failure

build-base-sno.yaml:175 — Pull secret cleanup only runs in publish-snapshot with if: success(). When validation fails, publish is skipped — cleanup never runs. The pull secret stays on the runner indefinitely. On shared baremetal infrastructure this is a credential leak.

Fix: Add if: always() cleanup in every job that writes the pull secret.

MAJOR — Dead push trigger

build-base-sno.yaml:23 — The push trigger (branches: [main], paths: [scripts/**]) is dead. build-snapshot has if: github.event_name == workflow_dispatch, so everything is skipped on push events. Wastes runner allocation.

Fix: Remove the push trigger. This workflow is manual-only (requires a source clone ID input).

MAJOR — Must use OSAC baremetal infrastructure

This workflow boots VMs via libvirt, runs cluster-tool commands, and interacts with live OpenShift clusters — all of this requires bare metal hardware. OSAC already has self-hosted GitHub Actions runners on baremetal machines, registered at the org level with the osac-ci label. These machines have everything pre-installed via machine-init.sh (libvirt, qemu-kvm, podman, haproxy, cluster-tool, dnsmasq, Vault CLI, helm, oc, etc.) and are connected to Vault for secrets management (pull-secret, registry auth via AppRole).

The workflow should:

  1. Use runs-on: [self-hosted, osac-ci] to target these machines (matching existing E2E workflow convention)
  2. Fetch secrets from Vault via AppRole instead of relying on GitHub Actions secrets (see fetch-secrets composite action in osac-test-infra)
  3. Use flock-based concurrency for flavor operations (exclusive lock for push, shared lock for boot) to prevent conflicts with concurrent CI runs on the same machine
  4. Leverage pre-installed tools on the runners rather than installing them at runtime (helm, oc, cluster-tool are all set up by machine-init.sh)

MINOR — destroy || true silently swallows cleanup failures

build-base-sno.yaml:151 — If destroy fails, leaked VMs/networks/disk overlays accumulate on the baremetal runner. The if: always() guard already ensures the step runs; || true is unnecessary suppression.

Fix: Remove || true. Let cleanup failures be visible.

NIT — No input validation

Neither ocp_version nor source inputs have pattern validation. Typos produce confusing failures deep in the pipeline.


Pipeline design (build → validate → publish) is the right approach, but the critical issue of snapshot_base.py pushing before validation undermines it. The pull secret handling also needs fixing for multi-runner environments.

Signed-off-by: omer-vishlitzky <omer.vishlitzky@gmail.com>
Assisted-by: Claude Code <noreply@anthropic.com>
…dation, timeouts

Signed-off-by: omer-vishlitzky <omer.vishlitzky@gmail.com>
Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: omer-vishlitzky <omer.vishlitzky@gmail.com>
Assisted-by: Claude Code <noreply@anthropic.com>
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

Superseded by #38 (unified snapshot pipeline PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant