Rebuild published UBI images nightly to pick up Iron Bank base fixes

[mcosgriff]

Problem

Iron Bank republishes ubi9-minimal (and the traefik image we build on) under
the same mutable tag as Red Hat ships fixes. Our published *-ubi images are
built once at release time, so their CVE count only grows until the next
COSMOS release, even when a patched base is already available.

Proposal

A scheduled workflow (nightly, plus workflow_dispatch) that, for each
selected release:

1. Checks out the release's git tag
2. Rebuilds the 11 *-ubi images with --pull against the current Iron Bank base
3. Scans the result with Trivy
4. Pushes to Docker Hub and ghcr.io

Inputs (dispatch) / defaults (schedule):

• versions: explicit list (6.10.0,6.9.2) or supported (latest patch of
  the last N minors, N configurable)
• ubi_tag: override the base tag, default = the tag in that release's .env
• force: rebuild even if the base digest is unchanged

Open questions / requirements

What to rebuild

• Skip a release when its base digest (ubi9-minimal, traefik, QuestDB
  RHEL) is unchanged since the last rebuild; record the digest in an
  org.opencontainers.image.base.digest label and compare against it.
  Package updates inside the Dockerfiles can lower the CVE count even on an
  unchanged base, so force (or a periodic forced run) still has value
• Older releases pin an older minor (e.g. OPENC3_UBI_TAG=9.6), which
  Iron Bank may no longer refresh. Decide whether to rebuild on the
  release's minor only, or allow bumping to the current minor (a base OS
  change on a published version)
• Older tags carry older build scripts: use each tag's own
  build_multi_arch.sh, or main's script against the tag's sources

Tagging

• Overwrite vX.Y.Z-ubi in place, and/or add an immutable
  vX.Y.Z-ubi-YYYYMMDD so users pinned to a rebuild can stay on it
• Confirm Docker Hub and ghcr.io allow overwriting these tags (disable tag
  immutability if enabled)
• Move latest only when the rebuilt version is the current latest
• Confirm cleanup-ghcr.yml doesn't delete the superseded manifests that
  digest-pinned users still pull

Verification before push

• Trivy must show no new CRITICAL/HIGH vs the currently published image;
  don't push a regression
• Smoke test: containers start and the stack comes up healthy

After push

• Regenerate the Trivy results and SBOMs attached to the GitHub release
  (post_release_trivy.yml)
• Summarize per release: old vs new base digest, CVE delta, tags pushed

Operational

• Iron Bank login currently uses a personal account's CLI token; move it
  to a service account before a scheduled job depends on it
• concurrency group so it can't overlap a release run
• Notify on failure (a stuck nightly means images silently go stale)

Enterprise

COSMOS Enterprise UBI images build on these core images, so they need the same
treatment:

• Equivalent nightly rebuild in cosmos-enterprise, triggered after the core
  rebuild pushes (repository_dispatch or workflow_run), not on its own
  schedule, so it always builds on the fresh core images
• Same version selection, tagging, Trivy gate and release-asset refresh
• Enterprise-only images with their own external bases (e.g. Keycloak) get
  --pull too

[crawlchat]

This reads like a design proposal rather than a question, so I didn't check the Question label. Here is what the repo already covers for your checklist. I ran out of search budget before checking post_release_trivy.yml triggers, the traefik Dockerfile, and older tags' scripts.

Reusable pieces already in the repo

• Smoke test and Trivy scan: build_ubi.yml runs ./openc3.sh build-ubi, then run-ubi, sleeps 180s, and fails on Restarting or unhealthy containers. It then scans all 11 images with trivy image --config trivy.yaml --image-src docker. You can lift both into the nightly.
• Iron Bank login: build_ubi.yml and release.yml both run docker login registry1.dso.mil -u jmthomas with secrets.IRONBANK_REGISTRY_CLI. That is the personal-account dependency you want to remove, and it appears in two workflows.
• Release build: release.yml runs ./build_multi_arch.sh ubi with OPENC3_UPDATE_LATEST taken from the update_latest input.

Findings on the open questions

• ubi_tag override: this should work without script changes. scripts/release/build_multi_arch.sh loads .env as defaults only, so an exported OPENC3_UBI_TAG wins. The default today is 9.6 in .env and in the Dockerfile ARGs.
• latest: the script only pushes latest when OPENC3_UPDATE_LATEST == true. The nightly can set that only when the rebuilt version is the current latest.
• Tagging: the script pushes :${OPENC3_RELEASE_VERSION} to both OPENC3_REGISTRY and OPENC3_ENTERPRISE_REGISTRY in one build. An extra -YYYYMMDD tag would need another -t (or a docker buildx imagetools create retag after the scan).
• cleanup-ghcr.yml: it only targets openc3-*-buildcache with delete-untagged: true. It won't touch openc3-*-ubi release packages today. Overwriting a tag leaves the old manifest untagged, so never extend that cleanup to the release packages.
• Docker Hub and ghcr.io tag immutability: this is a registry setting, not visible in the repo. Someone with admin access needs to check it.

Gotchas I found

• Scan before push: the script builds with --push directly, so there is no gate before publishing. UBI is linux/amd64 only, so a --load, then Trivy, then push flow is feasible. It needs a script change or a separate nightly build path.
• --pull: I didn't see --pull in the part of the script I read (first 120 lines). It would need to be added to the retry_build calls.
• Build order: openc3-node/Dockerfile-ubi builds FROM ${OPENC3_REGISTRY}/${OPENC3_NAMESPACE}/openc3-ruby-ubi:${OPENC3_TAG}, so it pulls ruby from the registry. A gate-before-push flow must build ruby first and make it available to dependents, for example by pushing it first or using a local registry or builder context.
• Base digest: ruby, redis and buckets use OPENC3_UBI_REGISTRY/OPENC3_UBI_IMAGE:OPENC3_UBI_TAG. tsdb uses OPENC3_DEPENDENCY_REGISTRY/OPENC3_TSDB_IMAGE:<version>-rhel. The digest check has to cover both, plus traefik, whose Dockerfile I didn't inspect.
• Concurrency: build_ubi.yml groups by workflow and ref. The nightly needs a group shared with release.yml, which I didn't inspect for a concurrency block.

Still needed from the team

• Which tags' build_multi_arch.sh exist and work for older releases (I couldn't check).
• Whether post_release_trivy.yml can be workflow_call/workflow_dispatch-triggered per version. It has a UBI_IMAGES list around line 277, but I didn't read its on: block.
• The cosmos-enterprise side, which I can't see from this repo.