From 2300e4761fea701756ca8e05ad223cbe2b17c34b Mon Sep 17 00:00:00 2001 From: mrbobbytables Date: Fri, 11 Sep 2026 15:50:04 +0000 Subject: [PATCH] docs: describe Utah's current architecture, parity, and installer status Bring documentation in line with the implemented Hummingbird-only runtime package model, current contract closure, installer and offline payload, deferred parity debt packages, and Dakota-style quality gates. - Distinguish local implementation status (VM bootc disk and live ISO with offline OCI payload) from public distribution availability. - Update package parity table and documented deferred packages to match machine-readable manifests (58 Bluefin, 13 additions, 71 verified, 6 unavailable). - Clarify that Fedora and Rawhide are never runtime package sources and that Utah installs Hummingbird builds. - Document Dakota-style preflight, build, artifact verification, E2E VM/installer, and promotion/rollback quality gates. Assisted-by: Gemini 3.8 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables --- AGENTS.md | 6 +- Justfile | 5 +- README.md | 101 ++++++++++++++++++++++++++------ docs/building.md | 67 ++++++++++++++++++++- docs/skills/index.json | 6 +- docs/skills/index.md | 2 +- docs/skills/local-testing.md | 20 ++++--- docs/skills/package-contract.md | 16 ++--- packages/utah.toml | 2 +- 9 files changed, 178 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 92965df2..bb0205bc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,10 @@ # Utah — Agent Operating Contract Utah composes [Bluefin](https://projectbluefin.io) on the Fedora Hummingbird -`bootc-os` base. Experimental pre-alpha: it has a build, not users. Nothing is -published — no image, no ISO artifact, no installer. +`bootc-os` base. Experimental pre-alpha: it has a build, not users. No images or +release ISOs are publicly published yet; local implementation supports image +composition, bootc VM disk installation, and live ISO with embedded offline +bootc-installer. ## Read order diff --git a/Justfile b/Justfile index 6706129f..e2291b87 100644 --- a/Justfile +++ b/Justfile @@ -338,9 +338,8 @@ generate-bootable-image stream="testing": sync echo "Bootable disk ready: $disk" -# Build a single-architecture UEFI live ISO. This first slice proves the -# Utah live boot path; installer payload integration is intentionally the next -# ISO milestone. +# Build a single-architecture UEFI live ISO with an embedded bootc-installer +# and offline OCI payload. iso stream="testing" debug="0": #!/usr/bin/env bash set -euo pipefail diff --git a/README.md b/README.md index 1ffae678..800411b2 100644 --- a/README.md +++ b/README.md @@ -8,10 +8,11 @@ Bluefin built on Fedora Hummingbird. The more ... civilized murder machine. ![alt](https://github.com/user-attachments/assets/56428338-54a0-4376-a53b-5f02f8b101a1) **Experimental pre-alpha** — the image builds, boots, and reaches GDM in local -QEMU validation. No image has been published to a registry, there is no -installer payload, and no ISO has been released. Nothing here is ready to run on -a machine you care about. [Filing -issues](https://github.com/projectbluefin/utah/issues) is the whole point. +QEMU validation, and a single-architecture UEFI live ISO with an embedded +offline installer payload builds locally (`just iso`). However, no images have +been published to a public registry, and no official ISO release artifacts have +been published. Nothing here is ready to run on a machine you care about. +[Filing issues](https://github.com/projectbluefin/utah/issues) is the whole point. ## What it is @@ -21,6 +22,11 @@ bootable base and no desktop at all. Utah adds the desktop: Bluefin's package contract on top, and the GNOME 51 stack built from source because neither Hummingbird nor a Fedora release ships it. +Utah installs runtime packages strictly from its Hummingbird base and overlay +plus the pinned `utah-packages` factory repository. Fedora and Rawhide +repositories are never enabled at runtime (Fedora 44 is used strictly as a +builder-stage toolchain for kernel cache builds). + alt Two repositories, the way `common` and `brew` already work: @@ -45,6 +51,14 @@ kernel cache image gets built at all. **None of these are published yet.** The tags above describe what the pipeline is built to produce, not something you can pull today. +## Status: local implementation vs public availability + +| Area | Local implementation status | Public distribution status | +|---|---|---| +| **OCI image composition** | Implemented (`just build-ghcr utah testing main` or `just build-local`) | Not yet published to GHCR | +| **Installed VM execution** | Implemented (`just generate-bootable-image testing && just boot-vm`); reaches GDM and GNOME Shell | No pre-installed disk images distributed | +| **Live ISO & installer** | Implemented (`just iso testing && just boot-iso`); UEFI ISO with systemd-boot, embedded `bootc-installer` Flatpak, and offline OCI VFS payload | No ISO artifacts released | + ## Package parity with Bluefin `packages/bluefin.toml` is a byte-for-byte copy of Bluefin's `base.toml`, and CI @@ -53,35 +67,84 @@ being noticed later. | | count | | --- | --- | -| Bluefin contract installed | **61** | -| Utah additions (GNOME 51, desktop services) | 12 | -| Genuinely unavailable | **4** | +| Bluefin base contract installed | **58** | +| Utah additions (10 GNOME 51, 3 desktop services) | **13** | +| Total verified contract packages | **71** | +| Genuinely unavailable (deferred parity debt) | **6** | The install writes its resolved list to `/usr/share/utah/contract.txt` and the verify step asserts *that file*, so the two cannot disagree. - +### Documented unavailable packages (parity debt) + +Every package Bluefin ships that is absent from Utah's enabled runtime +repositories is tracked under `[unavailable]` in `packages/utah.toml`: + +- **`evolution-ews-core`** (#10) — Exchange Web Services plugin for Evolution. + Requires the entire evolution-data-server dependency closure (camel, + evolution-mail, libebook, libecal), which the factory does not yet build. +- **`firefox`** (#35) — Bluefin's desktop contract names the Firefox RPM, but + the factory only carries `mozjs140`. Utah ships Firefox via Flatpak in desktop + and live installer configurations instead. +- **`fish`** (utah-packages#19) — Upstream spec uses `%cargo_generate_buildrequires`, + expecting Rust crates as individual system RPMs that do not exist in the + buildroot. Needs a vendored/cargo-fetch rebuild. +- **`grub2-efi-x64-cdboot`** (#35) — Boot-media subpackage absent from enabled + repositories; Utah uses Hummingbird's bootable base and builds UEFI live media + with systemd-boot. +- **`pipewire-libs-extra`** (#35) — Absent from Hummingbird and utah-packages; + Bluefin pulls it from negativo17's `fedora-multimedia`, which Utah does not enable. +- **`zsh`** (utah-packages#21) — HTML docs build pulls texi2html, causing a file + conflict in the buildroot between Hummingbird's ruby4.0 and Fedora's ruby. + +## CI/CD pipeline and Dakota-style quality gates + +Utah adopts Dakota-style quality gates to validate every change before +promotion or publication: + +1. **Preflight contract gate** — `just check` validates manifests, workflow + mappings, download integrity checksums, desktop contract rules, and extension + metadata. `just check-parity` verifies `packages/bluefin.toml` against + upstream Bluefin, and `just check-repos` confirms package availability in + runtime repositories before any container build runs. +2. **Build gate** — `build.yml` splits the build matrix into `build_main` + (pristine Hummingbird base) and `build_kernel` (built on the content-hashed + kernel cache image). Builds run with read-only layer caching on PRs. +3. **Artifact verification gate** — Images are validated with in-image verifiers: + `verify-rpm-contract.py` checks that every package resolved in + `/usr/share/utah/contract.txt` is installed; `verify-desktop-contract.py` + verifies Utah branding, os-release identity (`NAME=Utah`, `ID=utah`), and + service presets; `verify-gnome-extensions.py` verifies GNOME 51 compatibility + for all bundled extensions. Signed artifacts carry cosign keyless signatures, + SBOMs, and SLSA provenance. +4. **End-to-end VM & installer testing gate** — `just generate-bootable-image` + and `just boot-vm` exercise full `bootc install to-disk` and QEMU boot to GDM. + `just iso` and `just boot-iso` validate live boot and offline installation. + CI runs `.github/workflows/post-testing-e2e.yml` on testing builds before + advancing `:testing`. +5. **Promotion and rollback lifecycle** — Passing `:testing` images are promoted + to `:stable` via `.github/workflows/promote-testing-to-main.yml`. Pushes to + `main` trigger `.github/workflows/sync-main-to-testing.yml` to keep the + integration branch in sync. Deployed systems can roll back at any time with + standard `bootc rollback`. ## Known gaps This is the honest list, and it is why the label above says pre-alpha. -- **Nothing is published.** No image, no ISO artifact, no installer. A live - ISO builds locally (`just iso`); installer payload integration is the next - ISO milestone. +- **Public distribution is pending.** No OCI image or ISO artifact has been + published to a public registry or release download. The local build, VM boot, + and live ISO with offline installer payload are fully implemented. - **The NVIDIA and gaming flavors are unproven.** The OGC kernel compiles with `sched_ext` and `binderfs` genuinely enabled, and the NVIDIA open module compiles for the base kernel. The module against the OGC kernel, the driver installer flags, and the flavored builds pulling the kernel cache image have not yet all passed in one run. -- **`pipewire-libs-extra` is missing.** It was never a Fedora package; it exists - only in negativo17's `fedora-multimedia`, which Bluefin enables for its whole - install. -- **Codec support differs.** Twelve `[multimedia_overrides]` names are packages - Fedora already ships and Bluefin *replaces* with negativo17 builds. Utah - installs Fedora's. Nothing is absent from the image; hardware-accelerated - codecs are what differ. `utah-packages` already builds several of them, so - this closes when Utah consumes that overlay. +- **Hardware-accelerated codec support differs.** Twelve `[multimedia_overrides]` + names are packages Hummingbird already ships and Bluefin *replaces* with + negativo17 builds. Utah installs Hummingbird's builds of those packages; + Fedora and negativo17 repositories are never enabled at runtime. Hardware-accelerated + codecs differ until Utah consumes the `utah-packages` overlay for them. - **The image is still pre-alpha.** The digest-pinned `utah-packages` OCI repository is consumed and the local QEMU image reaches GDM and GNOME Shell. - **CUDA is deliberately excluded** — 7.68 GB installed. Use the NVIDIA diff --git a/docs/building.md b/docs/building.md index 56ef36b0..a60b869f 100644 --- a/docs/building.md +++ b/docs/building.md @@ -1,7 +1,8 @@ # Building Utah -Contributor quickstart: the local build loop. The deep documentation lives in -the skills linked at the bottom; the README is the user-facing page. +Contributor quickstart: the local build loop, testing procedures, Dakota-style +quality gates, and artifact verification. The deep documentation lives in the +skills linked at the bottom; the README is the user-facing page. ## Build locally @@ -13,6 +14,13 @@ just boot-vm # Optional local diagnostics over SSH (never use for a published image): ENABLE_SSHD=1 just build-ghcr utah testing main + +# Compose against a local package repository image: +just build-local testing localhost/utah-packages:local-merged + +# Build the live ISO with offline installer payload: +just iso testing +just boot-iso ``` The image is tagged `localhost/utah:testing`. `generate-bootable-image` uses @@ -25,6 +33,61 @@ noVNC and SSH ports; SSH is available only when the image was built with unified-storage's registry repull; published images omit that argument and keep the service enabled. +## Dakota-style quality gates + +Utah adopts Dakota's staged validation topology to ensure reliability and +parity before artifacts are published or promoted: + +### 1. Preflight contract gate +Static checks run before expensive builds: +- `just check` — validates manifests, workflow step mappings, download integrity, + extension metadata, desktop contract syntax, and skill catalog freshness. +- `just check-parity` — asserts byte-for-byte identity between `packages/bluefin.toml` + and upstream Bluefin `base.toml`. +- `just check-repos` — verifies every contract package is present in enabled + runtime repositories (Hummingbird base and `utah-packages` factory). + +### 2. Image and kernel build gate +- Split matrix: `build_main` builds directly on the clean Hummingbird base; + flavored builds (`build_kernel`) build on the content-hashed kernel cache image + (`scripts/kernel-cache-tag.sh`), building on demand on cache miss. +- Layer discipline and caching: OCI layer caches are read-only on PRs and + updated only on pushes to `testing`. + +### 3. Artifact verification +- **RPM contract**: `scripts/verify-rpm-contract.py` verifies the exact package + list recorded at install time in `/usr/share/utah/contract.txt`. +- **Desktop contract**: `scripts/verify-desktop-contract.py` validates identity + (`NAME=Utah`, `ID=utah`, `VERSION_CODENAME=Utahraptor`), branding assets, + dconf profiles, and service presets (`gdm.service`, `systemd-resolved`, etc.). +- **GNOME extensions**: `scripts/verify-gnome-extensions.py` ensures all 9 bundled + extensions declare GNOME 51 support in `metadata.json`. +- **Download integrity**: `scripts/check-download-integrity.py` ensures all + executable downloads (uupd, NVIDIA drivers, Flatpaks) are pinned by checksum. +- **Supply chain**: CI generates cosign keyless signatures, SBOMs, and provenance + attestations via `projectbluefin/actions@v1`. + +### 4. End-to-end VM and installer testing +- **Disk VM validation**: `just generate-bootable-image testing && just boot-vm` + performs a full `bootc install to-disk` to a sparse disk image and boots QEMU + to verify graphical target and GDM. +- **Live ISO & offline installer validation**: `just iso testing && just boot-iso` + assembles a UEFI live ISO with systemd-boot, embedding the `bootc-installer` + Flatpak bundle (`org.bootcinstaller.Installer`) and the target image in a VFS + `containers-storage` graphroot for offline installation. Booting validates the + `UTAH_LIVE_READY` serial marker and offline installer execution. +- **CI E2E gate**: `.github/workflows/post-testing-e2e.yml` verifies built + digest artifacts before advancing the `:testing` tag. + +### 5. Promotion and rollback lifecycle +- **Stream promotion**: Images land first on `:testing`. After passing post-testing + E2E validation, `.github/workflows/promote-testing-to-main.yml` promotes testing + to `main`, updating `:stable` tags. +- **Branch synchronization**: `.github/workflows/sync-main-to-testing.yml` syncs + protected `main` back into `testing` on every merge. +- **Host rollbacks**: Deployed client systems roll back atomically via native + `bootc rollback`. + ## Deep documentation | Topic | Skill | diff --git a/docs/skills/index.json b/docs/skills/index.json index 9a380aa0..33d69865 100644 --- a/docs/skills/index.json +++ b/docs/skills/index.json @@ -1,5 +1,5 @@ { - "generated_at": "2026-09-05", + "generated_at": "2026-09-11", "schema_version": "1.0", "skills": [ { @@ -107,7 +107,7 @@ ], "description": "Local validation loop: build-ghcr, bootc install to-disk, QEMU/noVNC boot, live ISO. Use when validating changes end-to-end or debugging boot, GDM, or live-session failures.", "version": "1.0", - "last_updated": "2026-09-05", + "last_updated": "2026-09-11", "doc_type": "runbook" }, { @@ -125,7 +125,7 @@ ], "description": "Bluefin parity contract: verbatim bluefin.toml, utah.toml overlay, [unavailable] rules, repository policy. Use when adding, removing, or debugging packages or parity/check-repos failures.", "version": "1.0", - "last_updated": "2026-09-05", + "last_updated": "2026-09-11", "doc_type": "policy" }, { diff --git a/docs/skills/index.md b/docs/skills/index.md index 2864a0b1..32e7fa05 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -3,7 +3,7 @@ This file is a human-readable mirror of `index.json`. Both are generated by `scripts/generate_skill_index.py` — do not hand-edit either file. -Generated: 2026-09-05 · schema 1.0 · 8 skills +Generated: 2026-09-11 · schema 1.0 · 8 skills | id | category | status | one-line purpose | |---|---|---|---| diff --git a/docs/skills/local-testing.md b/docs/skills/local-testing.md index 67755c90..88cc8e23 100644 --- a/docs/skills/local-testing.md +++ b/docs/skills/local-testing.md @@ -1,7 +1,7 @@ --- name: local-testing version: "1.0" -last_updated: "2026-09-05" +last_updated: "2026-09-11" id: local-testing one_line_purpose: Build, install, and boot Utah locally in a VM or live ISO. entry_point: docs/skills/local-testing.md @@ -106,12 +106,14 @@ The local OCI ref is not available from the guest's localhost registry, so unified-storage service is skipped instead of retrying its registry repull forever. Published images omit that argument and keep the service enabled. -## Live ISO (initial bring-up) +## Live ISO (offline installer environment) -The first ISO slice reuses Utah's own kernel, dracut-live, and GNOME image. -`just iso` builds a single-architecture UEFI live ISO; this first slice proves -the Utah live boot path, and installer payload integration is intentionally -the next ISO milestone (comment above `iso` in `Justfile`): +The live ISO reuses Utah's own kernel, dracut-live, and GNOME image. `just iso` +builds a single-architecture UEFI live ISO that embeds the bootc-installer +Flatpak bundle (`org.bootcinstaller.Installer`) and the target OCI image in a +VFS `containers-storage` graphroot for offline installation (Dakota's +offline-payload design adapted for Utah's conventional bootc base; recipe +comment above `iso` in `Justfile` and `iso/scripts/build-iso.sh`): ```bash just iso testing @@ -120,9 +122,9 @@ just iso testing 1 # optional live-session SSH diagnostics (debug=1) ``` The result is `output/utah-live.iso`, assembled with systemd-boot, a -`UTAH_LIVE` dmsquash-live root, and a serial `UTAH_LIVE_READY` marker. It is -intended to prove live desktop boot first; bootc-installer/offline payload -integration is the next ISO milestone. `just boot-iso` boots it with +`UTAH_LIVE` dmsquash-live root, and a serial `UTAH_LIVE_READY` marker. The +offline installer payload is fully integrated locally; public release ISO +artifacts have not yet been published. `just boot-iso` boots it with QEMU-for-Docker and exposes the noVNC console at the printed URL (comment above `boot-iso` in `Justfile`), with TPM, UEFI, and `-snapshot` so nothing persists. diff --git a/docs/skills/package-contract.md b/docs/skills/package-contract.md index d19d53f3..a653630c 100644 --- a/docs/skills/package-contract.md +++ b/docs/skills/package-contract.md @@ -1,7 +1,7 @@ --- name: package-contract version: "1.0" -last_updated: "2026-09-05" +last_updated: "2026-09-11" id: package-contract one_line_purpose: Maintain Bluefin package parity and Utah's overlay manifest. entry_point: docs/skills/package-contract.md @@ -57,9 +57,10 @@ Bluefin's `[multimedia_overrides]` (twelve names: mesa-libGL, mesa-vulkan-drivers, libva, intel-mediasdk, libheif and friends) are **not** extra packages. They are the same names Fedora already ships, which Bluefin *replaces* with negativo17 builds by enabling `fedora-multimedia`. Utah does -not enable that repository, so it installs Fedora's builds instead. Nothing -is absent from the image; what differs is which build it carries, and the -practical consequence is hardware-accelerated codec support. +not enable that repository, so it installs Hummingbird's builds instead. Fedora +repositories are never enabled at runtime. Nothing is absent from the image; +what differs is which build it carries, and the practical consequence is +hardware-accelerated codec support. That is why they are absent from the contract rather than listed under `[unavailable]`: recording them as missing would be wrong (a source does @@ -114,9 +115,10 @@ about installation. - Drift in `packages/bluefin.toml` from upstream is a CI failure (`just check-parity`). -Current counts, per the README "Package parity" section: 61 Bluefin contract -packages installed, 12 Utah additions (GNOME 51, desktop services), 4 -genuinely unavailable. +Current counts, per the README "Package parity" section: 58 Bluefin contract +packages installed, 13 Utah additions (10 GNOME 51, 3 desktop services; 71 total +verified), 6 genuinely unavailable (deferred parity debt: evolution-ews-core, +firefox, fish, grub2-efi-x64-cdboot, pipewire-libs-extra, zsh). ## Verification diff --git a/packages/utah.toml b/packages/utah.toml index dbc79677..a245e544 100644 --- a/packages/utah.toml +++ b/packages/utah.toml @@ -16,7 +16,7 @@ # intel-mediasdk, libheif and friends -- are not extra packages. They are the # same package names Fedora already ships, which Bluefin *replaces* with # negativo17 builds by enabling fedora-multimedia. Utah does not enable that -# repository, so it installs Fedora's builds of those names instead. Nothing +# repository, so it installs Hummingbird's builds of those names instead. Nothing # is missing from the image; what differs is which build of them it carries, # and the practical consequence is hardware-accelerated codec support. #