Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,8 @@ concurrency:
cancel-in-progress: true

jobs:
# Cheap, network-only gate. A contract package that none of Utah's
# repositories provide used to
# surface as an opaque `exit status 71` from the image build; catching it here
# costs seconds instead of the full matrix.
# Resolve the complete transaction against the same pinned inputs as the
# image before starting the flavor matrix, including extension build tools.
contract:
runs-on: ubuntu-24.04
permissions:
Expand All @@ -34,8 +32,6 @@ jobs:
submodules: recursive
- name: Install just
uses: extractions/setup-just@f8a3cce218d9f83db3a2ecd90e41ac3de6cdfd9b # v3
- name: Install zstandard
run: pip install --quiet zstandard
- name: Validate manifests
run: just check
- name: Check Bluefin parity
Expand Down
4 changes: 2 additions & 2 deletions Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ ARG BASE_IMAGE=quay.io/hummingbird-community/bootc-os:latest@sha256:c5539f9ed4d9
# Keep this pin in Utah so an image build is reproducible and can be reviewed
# against the exact package set it consumes.
ARG PACKAGE_IMAGE=ghcr.io/projectbluefin/utah-packages
ARG PACKAGE_IMAGE_SHA=sha256:2848c60d51fc6d75c3c89b246aad0e5ebf1fe84c5b7696203f02f14727bd158b
ARG PACKAGE_IMAGE_SHA=sha256:ca320b39b5f40bea9516f6f1c11e70d352c35f1c3d109b3aaf0816be007468f7
# CI keeps PACKAGE_IMAGE_SHA pinned. PACKAGE_IMAGE_REF supports a local image
# in containers-storage, where no registry digest is available.
ARG PACKAGE_IMAGE_REF=${PACKAGE_IMAGE}@${PACKAGE_IMAGE_SHA}
Expand Down Expand Up @@ -78,7 +78,7 @@ RUN for pair in install-packages.py:utah-install-packages \
configure-branding.sh:utah-configure-branding \
verify-desktop-contract.py:utah-verify-desktop-contract \
verify-gnome-extensions.py:utah-verify-gnome-extensions; do \
install -m 0755 "/tmp/utah-scripts/${pair%%:*}" "/usr/local/libexec/${pair##*:}" || exit 1; \
install -Dm 0755 "/tmp/utah-scripts/${pair%%:*}" "/usr/local/libexec/${pair##*:}" || exit 1; \
done && \
cp -a /tmp/utah-common/. / && \
cp -a /tmp/utah-bluefin/. / && \
Expand Down
3 changes: 2 additions & 1 deletion Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ check:
python3 -m py_compile scripts/check-repo-availability.py
python3 scripts/install-packages.py --check packages/bluefin.toml
python3 scripts/verify-rpm-contract.py --check packages/bluefin.toml
python3 -m unittest discover -s tests -p 'test_package_resolution.py'
grep -qE 'reusable-build\.yml@(v1|[0-9a-f]{40} # v1)$' .github/workflows/build.yml
test -f Containerfile.kernel
bash -n scripts/install-ogc-kernel.sh
Expand Down Expand Up @@ -105,7 +106,7 @@ check-desktop-contract image_ref="localhost/utah:testing":

# Fail fast when a contract package is in none of the repositories the image
# actually enables, instead of discovering it twenty minutes into a build.
# Checks names only -- it does not assert versions. Needs network access.
# Resolves dependencies on the pinned base and package image. Needs podman and network.
check-repos:
python3 scripts/check-repo-availability.py packages/bluefin.toml packages/utah.toml

Expand Down
6 changes: 3 additions & 3 deletions docs/skills/ci-workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ as a `docs/building.md` design bullet; now lives in this skill).

## contract: the cheap gate

`build.yml` opens with a network-only gate so a contract package that none of
`build.yml` opens with a container-based gate so a contract package that none of
Utah's repositories provide fails in seconds instead of surfacing as an
opaque `exit status 71` from the image build (comment,
`.github/workflows/build.yml`). It runs three checks:
Expand All @@ -46,8 +46,8 @@ opaque `exit status 71` from the image build (comment,
(`scripts/check_workflow_outputs.py`) and the ban on flavor literals in
workflows.
- `just check-parity` -- `packages/bluefin.toml` against Bluefin's upstream.
- `just check-repos` -- contract package names against the enabled
repositories.
- `just check-repos` -- the complete installation transaction against the
digest-pinned base and package repository, including extension build tools.

The same job resolves the flavor set and splits it in two by what each
flavor builds on -- `main` on the pristine Hummingbird base, the rest on the
Expand Down
5 changes: 5 additions & 0 deletions docs/skills/containerfile.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,11 @@ because a multi-source COPY cannot rename, and installed by name into
`utah-` prefix.
3. Run `just check`.

The destination directory may be absent in the Hummingbird base. Use
`install -Dm 0755` in the loop, retaining `${pair%%:*}` for the source and
`${pair##*:}` for the destination. `${pair##:*}` does not strip the source
name: it installs a filename containing the entire colon-separated pair.

## Clean and lint share a layer

Everything above writes build-time residue that bootc lint rejects: dnf logs
Expand Down
18 changes: 12 additions & 6 deletions docs/skills/package-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,18 @@ about installation.

## Failure semantics

- A contract package missing from every repository the image enables is a
**build failure**: the dnf transaction in `install-packages.py` exits
non-zero and the Containerfile RUN step fails. `just check-repos` exists to
fail fast on that case — names only, no versions — instead of discovering
it twenty minutes into a build (recipe comment, `Justfile`, `check-repos`;
needs network access).
- A contract package or dependency missing from the installation transaction
is a **build failure**. `just check-repos` reads the base and package-image
digests from `Containerfile`, copies the same repository configuration, and
runs `install-packages.py --resolve` inside that base. This includes the
release-specific Bluefin section, GNOME, services, and extension build tools.
It needs Podman and network access. A name lookup on GitHub Pages is not
evidence that the pinned OCI repository is complete or ABI-compatible.
The factory's leading metadata layer is checked against the pinned manifest
and blob hashes, then mounted for resolution without downloading RPMs. The
factory must copy the same repodata into the leading layer and payload layer.
DNF's `--assumeno` may return 1 for a valid declined transaction; the checker
requires a transaction summary and rejects dependency and repository errors.
- `[unavailable]` entries still present in the install set are a validation
error (`install-packages.py --check`).
- Drift in `packages/bluefin.toml` from upstream is a CI failure
Expand Down
31 changes: 15 additions & 16 deletions packages/utah.toml
Original file line number Diff line number Diff line change
Expand Up @@ -110,26 +110,25 @@ packages = [
# either a ruby exclusion in the buildroot or dropping the html subpackage.
#
# firefox
# Bluefin's desktop contract names Firefox, but this factory currently only
# carries mozjs140 (the embeddable JavaScript runtime), not the Firefox
# browser build. Keep this explicit until a Hummingbird-compatible Firefox
# recipe is added. Tracked by #35.
#
# grub2-efi-x64-cdboot
# This boot-media subpackage is not present in either Hummingbird's overlay
# or the Utah package repository. The image uses Hummingbird's bootable base
# and does not build installer media here. Tracked by #35.
#
# pipewire-libs-extra
# Not published by Hummingbird or utah-packages. Bluefin obtains it from
# negativo17's fedora-multimedia repository, which Utah does not enable.
# Tracked by #35.
# Utah installs org.mozilla.firefox as a Flatpak through the desktop
# contract. The RPM browser is deliberately excluded, while mozjs140 is
# retained for GNOME Shell. Tracked by #35 and utah-packages#112.
#
# anaconda-live, slitherer
# Utah's ISO uses bootc-installer. Both alternative installer packages need
# anaconda-webui, which the factory excludes because Cockpit authentication
# needs a working GHmac backend in Hummingbird's GLib. Slitherer also lacks
# its Qt WebEngine dependency closure. Match the factory's installer policy
# until that path is supported; tracked by #83.
#
# grub2-efi-x64-cdboot and pipewire-libs-extra are supplied by the pinned
# September 10 factory candidate and return to the install contract (#35).
packages = [
"anaconda-live",
"evolution-ews-core",
"firefox",
"fish",
"grub2-efi-x64-cdboot",
"pipewire-libs-extra",
"slitherer",
"zsh",
]

Expand Down
188 changes: 86 additions & 102 deletions scripts/check-repo-availability.py
Original file line number Diff line number Diff line change
@@ -1,124 +1,108 @@
#!/usr/bin/env python3
"""Check the package contract against the repositories Utah actually installs from.
"""Resolve Utah's real install transaction against its pinned OCI inputs.

Every package Utah claims parity on is looked up in the repodata of the same
repositories the image build enables: Hummingbird's own overlay and the
utah-packages factory. A package that is neither available nor listed under
[unavailable] in packages/utah.toml fails here, in the seconds-long preflight
job, instead of twenty minutes into a container build.

Checking Rawhide instead would be actively misleading: Rawhide has moved to
OpenSSL 4 while the Hummingbird base pins 3.5.6, so a package being present
in Rawhide says nothing about whether Utah can install it.
A name lookup against Pages cannot verify the digest in Containerfile, library
dependencies, or packages supplied only by the base image's RPM database.
Use the same installer and repository files as the image, with --assumeno.
"""

from __future__ import annotations

import argparse
import hashlib
import io
import json
import re
import sys
import tomllib
import subprocess
import tarfile
import tempfile
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from pathlib import Path

# The same repositories the image installs from, in the same precedence order:
# Hummingbird's own overlay plus the utah-packages factory (its Pages mirror,
# since check-repos runs in CI before the OCI repo is copied into the image).
REPOS = {
"public-hummingbird": "https://packages.redhat.com/api/pulp-content/public-hummingbird/x86_64/",
"utah-packages": "https://projectbluefin.github.io/utah-packages/",
}


def section(path: Path, name: str) -> list[str]:
data = tomllib.loads(path.read_text())
return list(data.get(name, {}).get("packages", []))


def fetch(url: str) -> bytes:
with urllib.request.urlopen(url, timeout=120) as response:
return response.read()


def repo_package_names(base: str) -> set[str]:
repomd = ET.fromstring(fetch(base + "repodata/repomd.xml"))
ns = {"repo": "http://linux.duke.edu/metadata/repo"}
href = next(
location.get("href")
for data in repomd.findall("repo:data", ns)
if data.get("type") == "primary"
for location in data.findall("repo:location", ns)
)
raw = fetch(base + href)
if href.endswith(".zst"):
try:
import zstandard
except ModuleNotFoundError:
print(
"ERROR: python3-zstandard is required to read Rawhide repodata",
file=sys.stderr,
)
raise SystemExit(2)
stream = zstandard.ZstdDecompressor().stream_reader(io.BytesIO(raw))
else:
import gzip

stream = gzip.GzipFile(fileobj=io.BytesIO(raw))

names = set()
# Stream the ~16 MiB of metadata rather than holding a parse tree for
# 66k packages; only <name> at package scope matters here.
for line in io.TextIOWrapper(stream, encoding="utf-8", errors="replace"):
match = re.match(r"\s*<name>([^<]+)</name>", line)
if match:
names.add(match.group(1))
return names

def pinned_inputs(containerfile: Path) -> tuple[str, str]:
args = dict(re.findall(r"^ARG ([A-Z_]+)=(\S+)$", containerfile.read_text(), re.M))
base = args["BASE_IMAGE"]
packages = f"{args['PACKAGE_IMAGE']}@{args['PACKAGE_IMAGE_SHA']}"
for image in (base, packages):
if not re.fullmatch(r"[a-zA-Z0-9./:_-]+@sha256:[0-9a-f]{64}", image):
raise ValueError(f"Expected a digest-pinned image, got {image!r}")
return base, packages


def verified_bytes(raw: bytes, digest: str) -> bytes:
if digest != "sha256:" + hashlib.sha256(raw).hexdigest():
raise ValueError(f"Registry content does not match {digest}")
return raw


def repository_metadata(image: str, destination: Path) -> None:
registry, reference = image.split("/", 1)
if registry != "ghcr.io":
raise ValueError("The factory metadata reader currently supports ghcr.io images")
repository, digest = reference.split("@", 1)
query = urllib.parse.urlencode({"service": registry, "scope": f"repository:{repository}:pull"})
with urllib.request.urlopen(f"https://{registry}/token?{query}", timeout=120) as response:
token = json.load(response)["token"]

def fetch(kind: str, digest: str) -> bytes:
request = urllib.request.Request(
f"https://{registry}/v2/{repository}/{kind}/{digest}",
headers={"Authorization": f"Bearer {token}",
"Accept": "application/vnd.oci.image.manifest.v1+json"},
)
with urllib.request.urlopen(request, timeout=120) as response:
return verified_bytes(response.read(), digest)

manifest = json.loads(fetch("manifests", digest))
# The factory publishes repodata first, separately from the large RPM
# payload. Do not silently fall back to Pages or another image tag.
layer = manifest["layers"][0]
if layer["size"] > 64 * 1024 * 1024:
raise ValueError("Package image lacks a small leading metadata layer; republish with repodata first")
unpack_metadata(fetch("blobs", layer["digest"]), destination)


def unpack_metadata(raw: bytes, destination: Path) -> None:
with tarfile.open(fileobj=io.BytesIO(raw), mode="r:gz") as archive:
for entry in archive:
path = Path(entry.name)
if path.is_absolute() or ".." in path.parts:
raise ValueError(f"Unsafe metadata path: {entry.name}")
if entry.isdir():
continue
if not entry.isfile() or path.parts[:2] != ("repository", "repodata"):
raise ValueError(f"Unexpected entry in metadata layer: {entry.name}")
target = destination / Path(*path.parts[1:])
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(archive.extractfile(entry).read())
if not (destination / "repodata/repomd.xml").is_file():
raise ValueError("Pinned metadata layer contains no repomd.xml")


def main() -> int:
parser = argparse.ArgumentParser()
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("manifest", type=Path)
parser.add_argument("overlay", type=Path, nargs="?", default=None)
parser.add_argument("--engine", default="podman")
args = parser.parse_args()
root = Path(__file__).resolve().parent.parent
overlay = args.overlay or args.manifest.with_name("utah.toml")

unavailable = set(section(overlay, "unavailable"))
wanted = sorted(
set(section(args.manifest, "fedora"))
| set(section(overlay, "gnome"))
| set(section(overlay, "build"))
)
names: set[str] = set()
for label, base in REPOS.items():
found = repo_package_names(base)
print(f"{label}: {len(found)} binary packages")
names |= found

missing = [pkg for pkg in wanted if pkg not in names and pkg not in unavailable]
resolved = sorted(pkg for pkg in unavailable if pkg in names)

for pkg in resolved:
print(f"NOTE: {pkg} is now available and can be removed from [unavailable]")
if missing:
print(
f"ERROR: {len(missing)} contract packages are in none of the configured repositories.",
file=sys.stderr,
)
print(
"Add each to [unavailable] in packages/utah.toml with a tracking "
"issue, or fix the name:",
file=sys.stderr,
)
for pkg in missing:
print(f" - {pkg}", file=sys.stderr)
return 1
print(
f"All {len(wanted) - len(unavailable)} contract packages are available "
f"({len(unavailable)} documented as unavailable)."
)
return 0
base, packages = pinned_inputs(root / "Containerfile")
print(f"Checking package repository {packages} on {base}", flush=True)
with tempfile.TemporaryDirectory(prefix="utah-repodata-") as tmp:
repository_metadata(packages, Path(tmp))
return subprocess.run([
args.engine, "run", "--rm", "--platform", "linux/amd64",
"-v", f"{tmp}:/etc/utah-packages:ro,Z",
"-v", f"{root / 'packages'}:/etc/yum.repos.d:ro,Z",
"-v", f"{args.manifest.resolve()}:/tmp/bluefin.toml:ro,Z",
"-v", f"{overlay.resolve()}:/tmp/utah.toml:ro,Z",
"-v", f"{root / 'scripts/install-packages.py'}:/tmp/install-packages.py:ro,Z",
base, "python3", "/tmp/install-packages.py", "--resolve",
"/tmp/bluefin.toml", "/tmp/utah.toml",
], check=False).returncode


if __name__ == "__main__":
Expand Down
Loading
Loading