Canonical interface for driving the lab: every supported operation is one
argo submit --from workflowtemplate/<name> [-p k=v] (usually via a just
recipe). Templates live in argo/workflow-templates/ and ArgoCD testing-lab
reconciles them to namespace argo; CronWorkflows live in manifests/
(testing-lab-infra). Prefer the top-level pipelines; supporting templates are
called through templateRef. Update this file in the same commit as any
template add or rename.
- Pipelines
- KubeStellar Workflows
- Supporting Templates
- Distributed Build/RE Grid
- Cache Warming (Pollers)
- Nightly and Maintenance Schedules
- Priority Classes
- Resource Profiles
- Purpose: Run Dakota suites directly inside the published bootc OCI image using a container-only fan-out model.
- VM boundary: This active QA pipeline has no containerDisk build, VM boot, reboot, or SSH stage.
- Current coverage:
run-container-testsprovides the container-only suite fan-out, including the nested systemd/Wayland session used by qecore-headless. This lane provides image and GUI-session evidence, but not VM boot or reboot evidence. - Parameters:
image,image-tag,image-digest,suites,variant,branch,testsuite-branch,testsuite-repo. - DAG: parallel
test-laneitems (smoke,common,developer,software,system) throughrun-container-tests, filtered bysuites. - Just recipe:
run-dakota-qa. - PR review: use the Dakota PR review skill. Build the exact PR SHA first, then run smoke and required E2E suites against the resulting image. If lab validation identifies a scoped PR defect, repair the PR branch, rebuild from its new SHA, rerun E2E, and merge directly only after a fresh pass.
- Purpose: Direct kernel boot verification for Dakota images in a privileged pod.
Unpacks image layers directly into an ext4 raw disk via
mkfs.ext4 -d(no loop devices or bootc install), injects root SSH authorized keys, boots the guest kernel under QEMU/KVM with host CPU passthrough (-machine q35,accel=kvm -cpu host), and executes in-VM runtime evidence commands over SSH. - When to use: Validating image contents, kernel boot, systemd initialization,
gVisor (
runsc),krun/libkrunmicroVMs, and userspace filesystem tooling (fuse2fs,gocryptfs) against published or PR candidate Dakota images. - Parameters:
image(default192.168.1.102:30500/dakota:testing),commands(default:/etc/os-release,uname -r,systemctl is-system-running,systemctl --failed,bootc status). - Admission & Resources: Admitted through the
dakota-vm-bootsemaphore inmanifests/workflow-semaphores.yaml. Requests 6 CPUs, 12Gi memory, 40Gi ephemeral-storage (limits 8 CPUs, 14Gi memory, 80Gi ephemeral-storage), hostPath/dev, and emptyDir/work(80Gi). - Outputs: Captured evidence log (
evidence), serial log tail (serial-tail), and pod logs. - Just recipe:
run-dakota-vm-boot [image="..."].
- Purpose: The actual BuildStream compile step for Dakota — builds
oci/bluefin.bstand pushes the image to the local Zot registry under the tag given byimage-tag(defaulttesting). NVIDIA variants are built non-blocking and pushed with the same tag. - Parameters:
repo,ref,commit-sha,image-tag(defaulttesting),registry,variants. - Distribution: remote execution on BuildGrid is mandatory. A fresh USB4
upobservation and a Ready BuildGrid worker are required on bothghostandexo-0, plus a Ready BuildGrid controller, before admission. Cache-only, Ethernet-backed, automatic fallback, local-sandbox execution, and remote-cache-only execution are failures, not alternatives. Scheduler-driven placement selects the coordinator; no task pins it to a node. - Source fidelity: builds the exact dakota commit unmodified. The workflow
applies no element, junction, or patch-queue changes; upstream GNOME
reccdefaults apply when the pinned gnome-build-meta declares them. - Capacity: the coordinator keeps
scheduler.builders: 12element actions in flight withmax-jobs: 32each; GNOME recc fans each element out into compile actions. All actions queue in BuildGrid and run on theworkerDaemonSet (one per node,CONCURRENT_JOBS: 32slots each). Thebst-buildsemaphore admits two pipelines at once; each runs at most one coordinator per node, so lanes ×buildersstays below one node's slots and element actions cannot starve their own compiles. A new node adds capacity with no config change. The workflow verifies its generated remote-execution configuration before it invokes BuildStream. - Coordinator: each variant runs the shared
bst-build-reWorkflowTemplate (argo/workflow-templates/bst-build-re.yaml,templateReftemplatebuild). It is acontainerSet:main(bst2) builds and checks the OCI layout out onto the pod's CAS volume;publish(digest-pinnedghcr.io/projectbluefin/skopeo) then runs oneskopeo copyto Zot. A failed build never startspublish. Required per-workflow pod anti-affinity keeps one coordinator per node, sovariants=allruns in two waves. - Retry:
bst-build-reretries once (retryPolicy: Always) unlessmainexits 3 (expression: lastRetry.exitCode != "3").mainrunsbst buildthroughbst-build.shfrom thebuildstream-remote-cacheConfigMap, which exits 3 when an element's own build failed and BuildStream cached the failed artifact: the retry would fail the same way after ~20 minutes of pulls. Grid, CAS, network, and pod failures keep bst's exit code (255) and are retried.build.retry-failed: truerebuilds instead of replaying a cached failure. See BuildStream: Failed builds and retries. - Priority:
priorityClassName: bst-buildkeeps the coordinator ahead of short-lived lab test workloads. - Who triggers it automatically: the
dakota-commit-pollerCronWorkflow through the sharedbst-commit-pollertemplate (see Cache Warming). The poller resolves the current GitHub SHA fordakota:testingand passes that exact commit into the local BuildStream run, so the lab build checks out the same source revision that GitHub is building instead of drifting to a later branch tip.
Distributed-gate rule: A Dakota PR build is valid only when a fresh remote-execution run completes for the exact PR head SHA and its pushed registry image is verified. Local, cache-only, or fallback builds are diagnostic evidence and do not satisfy the distributed gate. Inspect failed child nodes even when the Argo parent phase is successful; classify BuildGrid controller/worker and Buildbarn storage/DNS failures as infrastructure blockers and repair the lab before retrying.
- Purpose: Reusable, independent single-lane lifecycle for immutable local
Zot candidates and digest-preserving promotion to
:testing. - Preflight: rejects reused
candidate-<commit-sha>tags and fails when the Zot data filesystem has less than 20 GiB or 10% free. - Integrity: resolves the expected digest, fetches and hashes the raw
manifest, copies that digest to
:testing, and verifies the target digest. - Evidence: attaches and rediscovers
application/vnd.projectbluefin.lab.promotion-evidence.v1+jsonwith ORAS. - Authentication: optionally mounts
zot-writer-authwhile anonymous writes remain active; the secret becomes required at the Zot auth activation gate. - Just recipe:
run-zot-promotion. See Zot candidate promotion.
- Purpose: BuildStream compile pipeline for
oci/bluefin-server-image.bst(the whole release set, built with the fixed dev keys) and push to local Zot asbluefin-server-image:latest. - Coordinator: the same
bst-build-retemplate as dakota, withdeadline: 10800(3h per pod; workflowactiveDeadlineSeconds: 28800), apod-patchthat adds thegithub-tokenenv and thebluefin-server-dev-boot-keysSecret mount (dakota passes none, so its pods carry neither Secret), andupstream-cache: https://cache.projectbluefin.io:11001(replacescache.freedesktop-sdk.iofor the toplevel project only). Retry, priority, and execution are identical to dakota. - Cache policy: CAS (
cache.storage-service) and action cache use the shared Buildbarn frontend (frontend.buildbarn.svc.cluster.local:8980); artifacts and sources are indexed inbb-remote-asset(:8984) and pushed there, with upstream caches as read-only fallbacks. Source caches override the project entries. Seemanifests/buildstream-remote-cache-config.yaml.
-
Purpose: ChairLift's behave + dogtail AT-SPI suite (
test/e2e/features) in a live GNOME Wayland session, plus an Enhanced Troubleshooting Goose-profile isolation check on x86_64. -
Parameters:
chairlift-repo,chairlift-branch,image(ghcr.io/projectbluefin/dakota),image-tag(testing),behave-tags(empty = every scenario),goose-isolation-check(true). -
Shape: a
golang:1.26init container clones the branch and runs ChairLift'smake cion the untouched checkout. It fetches golangci-lint v2.12.2, the version CI pins, by checksum, and records the result, which the verdict gates on. It then runsmake build-e2e schemas, stageschairliftandchairlift-wrapperthe way the Homebrew cask installs them, and builds two lane tools inside the module:navjson(the page/shortcut inventory the Go gate normally passes) andtroubleshootcheck.troubleshootcheckdrives the branch'sinternal/troubleshootandinternal/aistack: Setup, Agent Mode, the model alias path, the PATHCommandhands the launch, and a config reader. The runner then bootsrun-container-tests' nested systemd/GDM target: headless GNOME Shell drop-in, linger,lastchg, and the resolver bind./usr/share/chairliftis masked and/proc/cmdlineblanked, as the fixture harness does. Behave runs from a hash-pinned venv underqecore-headless --session-type wayland, withGDK_BACKEND=wayland, an absoluteWAYLAND_DISPLAY, andDISPLAYunset. Keys go through dogtail's Mutter RemoteDesktop backend. Xvfb is never used. -
Screenshots: ChairLift's own
capture_walkthrough.shruns under itswayland_session.sh(headless Mutter at 900x700, frames from Mutter's ScreenCast) inside the target, on a private bus beside the GDM session. Only the page PNGs are kept, inscreenshots/; the byproducts stay inscreenshots-raw/. -
Wayland traps: qecore hands the script gnome-session's environment, which has no
WAYLAND_DISPLAY; the script reads it fromsystemctl --user show-environment. Mutter's focus-stealing prevention leaves each new window unfocused, so ChairLift'senvironment.pyactivates it throughShell.Eval(needs--unsafe-mode). Dakota'sbrew-preinstall.serviceand brew timers are masked so they cannot hold Homebrew's locks during the Goose check. -
Goose check (end to end, ChairLift's own code throughout):
troubleshoot.Setupinstalls the tap,linux-mcp-server,cpio, and thegoose-linuxcask withPATH=/usr/bin:/binand nobrew shellenv, as a directchairliftlaunch would. The cask's barerpm2cpio | cpiopreflight exits 127 unless ChairLift puts Homebrew's bin on brew's PATH.aistack.Enableturns Agent Mode on, on the stock host. The check records thatbrew trustcovers thellmmanformula and notllmmanorg/tap.- The smallest Qwen3 is resolved through aistack's live resolver, pulled
with
aistack.PullModel(which must succeed on its own), and set as thebluefin-activealias. llmman reports the alias as the registry-qualifiedhf.co/<ref>. The lane also records/llmman/nodeandllmman ls. - Two launches run with GNOME Shell's own environment and a PATH without
Homebrew, in an app.slice scope. The first is a cold
chairlift --ask-bluefin. The second is the distro menu's exact command,bash -c '…/chairlift-wrapper --ask-bluefin'. Each must exit 0 with Goose running through llmman. The lane reads the environment Goose hands itsgoose servebackend (GOOSE_PATH_ROOT,XDG_CONFIG_HOME,GOOSE_PROVIDER=openai,OPENAI_HOSTon127.0.0.1:17434,GOOSE_MODEL); the two launches must agree on it. It also checks that the lock is in the profile and takes a screenshot. llmman launch goose -- run --text …, run in the same profile, must call alinux-tools__*tool andbluefin-knowledge__search_knowledge, read back fromdata/sessions/sessions.db. The answer is compared withsystemctl --failedfor 0.6B, 1.7B, and then 4B, and recorded. Answer quality does not gate the run.
Throughout, the profile's lock, data, state, and config must stay inside it, and only
linux-toolsandbluefin-knowledgemay be enabled. -
ChairLift runs as an app unit: every ChairLift step runs through
systemd-run --user --slice=app.slice, the way GNOME starts an app. The script itself runs in podman's exec cgroup, outside any logind session, where polkit answersauth_adminto the system Flatpak install Agent Mode does; inside an app unit, an activewheelmember getsyes. The test user joinswheel, as Bluefin's first user does.TasksMaxis lifted on the user slice and on the units, because brew's download threads hit the container-sized default. -
Verdicts: the run fails if any of these happens:
make cifails;- behave fails, or executes 0 scenarios;
- any engine check fails;
- the screenshot capture does not produce every page and the reset-group marker.
Outputs
result,failed-scenarios, andgoose-isolation(JSON). Everything under/tmp/results(behave log, JUnit, per-scenariochairlift.log,tree.txt,screen.png; Goosesetup.log,command-env.txt, lock evidence, config, screenshot) is persisted to ghost's result store under<workflow>/chairlift-wayland/. -
Submit: not deployed until merged; until then submit the template file as a Workflow (
kind: Workflow,generateName) withargo submit.
KubeStellar installation and upgrades are owned by the kubestellar-applications
ArgoCD parent Application. Each template takes wec-name (default ghost).
register-wec: registers the WEC with the its1 OCM hub and labels the ManagedClustername=<wec>. SAkubestellar-bootstrap(cluster-admin; klusterlet install writes CRDs).kubestellar-smoke-test: verifies BindingPolicy downsync and singleton status upsync via wds1 (kubeconfig-inclusterkey), then cleans up. SAkubestellar-bootstrap. Acceptance gate after any core upgrade.kubestellar-platform-verify: read-only gateverify-datasource → verify-query-surfaces → verify-controller-wiring → kubestellar-smoke-test, run withjust run-kubestellar-verify. Read-only checks usekubestellar-observability; the smoke task is the only one that creates resources.templateRefdoes not inherit workflow-level identity, so the smoke template declareskubestellar-bootstrapat template level.
| Template | Role |
|---|---|
image-poller |
Digest-comparison helpers (poll-digest, update-local). Flow: fetch upstream digest → compare with image-polling-digests → run downstream → persist digest only after downstream success. image-poll-dakota routes to dakota-qa-pipeline. |
run-container-tests |
Shared container-only runner for Dakota bootc images. Clones projectbluefin/testsuite, starts a Wayland session in the target OCI image, runs behave, attempts best-effort result publication when github-token is available, and writes a summary file for workflow outputs. Publication warnings do not change the suite exit status. |
bst-build-re |
Shared BuildStream coordinator (template build): builds one element on BuildGrid and publishes it to Zot. Callers hold the bst-build semaphore. |
bst-admission-gate |
detect-build-mode: the USB4/BuildGrid admission check each BST pipeline runs before its build tasks. |
bst-cache-warm |
Manual cache re-seed after cache loss: runs the Dakota and Bluefin Server build-warmup templates on the bst-cache-warm semaphore lane. |
Three cluster mechanisms cooperate on a distributed BuildStream build, each with one job:
| Mechanism | What it distributes | Used by |
|---|---|---|
k8s scheduler (no pin; coordinators prefer exo-0) |
BuildStream coordinator pods and OCI export/push | dakota-build-pipeline, bluefin-server-build-pipeline |
BuildGrid (buildgrid namespace) |
Remote-execution actions on buildbox-run-bubblewrap workers (private PID namespace, fresh /proc) |
dakota-build-pipeline, bluefin-server-build-pipeline |
Buildbarn (buildbarn namespace) |
CAS, action cache, and remote asset only; no execution | same lanes, plus BuildGrid workers' casd |
BuildGrid topology (manifests/buildgrid-*.yaml): controller
(controller.buildgrid.svc.cluster.local:50051, Execution/Operations), bots
(Remote Workers API), the stateless cas front, a Postgres database
StatefulSet, and a worker DaemonSet with one buildbox-casd + buildbox-worker
pod per node.
Buildbarn topology: 2 storage shards (spread with podAntiAffinity), 2
frontend replicas, and bb-remote-asset (manifests/buildbarn-*.yaml). Every
BST lane requires BuildGrid execution over a fresh USB4 link between ghost
and exo-0. If a link, worker, or action is unavailable, the workflow must fail
for repair; it must not use an Ethernet, local, or cache-only fallback.
| CronWorkflow | Interval | Triggers | Keeps warm |
|---|---|---|---|
dakota-commit-poller |
suspended | shared bst-commit-poller → dakota-build-pipeline when dakota:testing changes |
Dakota BuildStream cache/execution path; on-demand via just force-dakota-poll |
server-commit-poller |
every 15 min at :02 | shared bst-commit-poller → bluefin-server-build-pipeline when projectbluefin/server main changes |
Bluefin Server BuildStream cache/execution path |
image-poll-dakota |
every 10 min at :08 | custom digest DAG with run-qa=false |
Dakota testing digest freshness; daily QA runs at 03:00 UTC |
Dakota/Bluefin Server/BST lanes execute on BuildGrid and write caches to the shared Buildbarn frontend while leaving upstream mirrors read-only. Cold runs may fetch from upstream source origins, but cache writes stay in-cluster via Buildbarn.
Both commit pollers use the same USB4-gated BuildGrid remote-execution
contract as manual BST runs. bst-cache-warm (manual) re-seeds the Dakota and
Bluefin Server caches after cache loss on its own semaphore lane.
nightly-dakotadoes not warm anything — it's wired todakota-qa-pipeline(test runner against pre-built images), notdakota-build-pipeline(the actual compile step). The real Dakota cache-warming trigger isdakota-commit-poller. It must not be interpreted as proof of a green distributed build: the poller succeeds only when the remote BuildStream workflow, image export, registry push, and configured validation path succeed.
The commit CronWorkflows share one implementation. It compares
the source SHA, defers when every bst-build lane is busy and one more BST
workflow is already waiting (the ceiling is read from workflow-semaphores), invokes the
repository-specific build template, and writes the SHA only after that build
succeeds. Failed builds therefore remain eligible on the next poll.
The CronWorkflows pass force=false. For recovery after an out-of-band artifact
loss, just force-dakota-poll submits the Dakota CronWorkflow with force=true.
Force bypasses only the stored-SHA equality check: queue admission and the
bst-build semaphore still apply. A retried or resumed older workflow builds its
captured SHA, but skips the final state write if another successful workflow
advanced the stored SHA while it was running.
Current contract: image-poller must not update image-polling-digests
until run-pipeline.Succeeded. If the digest is written before QA passes, the
poller will treat the image as already seen and silently skip the failed lane on
the next cycle.
Bandwidth contract: image-poller resolves digests by inspecting
the upstream registry directly — never through the zot cache. Zot on-demand
sync copies manifest + all blobs on a tag read, so polling through zot pulled
every new multi-GB image even when QA was skipped.
| CronWorkflow | Time (UTC) | Pipeline | Parameters |
|---|---|---|---|
nightly-dakota |
03:00 | dakota-qa-pipeline |
image=ghcr.io/projectbluefin/dakota, image-tag=testing, suites=smoke,developer,system, variant=dakota |
orphan-vm-cleanup |
every 30 min | inline | Delete KubeVirt test VMs whose parent workflow is gone or terminal |
orphan-pod-gc |
every 30 min | inline | Delete ContainerStatusUnknown and stale Failed pods Argo podGC missed |
| PriorityClass | Value | Applied to |
|---|---|---|
bst-build |
1,500,000 (manifests/bst-build-priorityclass.yaml) |
Long BuildStream coordinators: dakota-build-pipeline, bluefin-server-build-pipeline |
lab-test-vm |
1,000,000, PreemptLowerPriority |
Explicit VM-backed KubeVirt test VMs (bluefin-server-boot-test) |
bst-build sits above lab-test-vm so a short-lived test VM never preempts a
multi-hour build and loses its progress.
Pod resource requests/limits used by workflow steps:
| Template / container | CPU req/limit | Memory req/limit |
|---|---|---|
run-container-tests |
2 / 4 | 4Gi / 8Gi |
run-chairlift-wayland-tests main |
2 / 4 | 4Gi / 8Gi |
run-chairlift-wayland-tests build-chairlift |
2 / 4 | 2Gi / 4Gi |
bst-build-re main |
2 / 4 | 4Gi / 8Gi |
bst-build-re publish |
1 / 16 | 256Mi / 2Gi |