GitOps k3s cluster on ghost + exo-0, built on CNCF projects.
Internal lab for feature development before sending PRs upstream. Three purposes,
all built on the 40 Gbps point-to-point USB4 link between ghost and exo-0
(thunderbolt0, 10.99.0.0/30). Policy routing (table 40, rule 5209) sends
cross-node pod traffic over the link; control-plane and DNS stay on Ethernet.
Clients on the LAN submit work as Argo Workflows (http://192.168.1.102:32746).
| Purpose | What runs | USB4 role | Status |
|---|---|---|---|
| BuildStream build farm | BuildGrid remote execution (buildgrid ns) with Buildbarn CAS/action cache (buildbarn ns), plus the dakota OCI pipeline: dakota-build-pipeline / bluefin-server-build-pipeline execute on BuildGrid, publish to the Zot registry (:30500), run Dakota QA and boot tests, and promote Zot candidates. Makes projectbluefin/dakota. |
Hard admission gate: remote-execution actions and CAS transfers run pod-to-pod over the link; no Ethernet or local fallback. | Live |
| Local LLM inference | llama.cpp on Vulkan (manifests/llm-d.yaml): exo-0 serves ornith worker and ghost serves ornith advisor; exo-0 NodePort 30800, ghost 30801, OpenAI-compatible /v1. |
Carries client and in-cluster traffic to the model servers. | Live |
| Distributed ffmpeg encoding | Plain Workflows from client tools (tools/farm.py in the video repos): frame-grid segments, parallel ffmpeg, concat -c copy join, ffprobe verify. |
Chunk exchange between the two nodes' pods. | Live per node; cross-node chunking over USB4 not built yet |
| Layer | Project | Role |
|---|---|---|
| Kubernetes | k3s | Local cluster |
| VM workloads | KubeVirt | Ephemeral boot-test VMs (bluefin-server-boot-test) |
| CI/CD | Argo Workflows | DAG pipeline orchestration |
| GitOps | Argo CD | Declarative cluster state from git |
| Control plane | KubeStellar | WDS/ITS control plane for the local lab |
| Administration | KubeStellar Console | Sole private cluster-admin and single-pane UI |
| Host | Role | Specs |
|---|---|---|
| ghost | k3s control-plane + KubeVirt compute | Ryzen AI MAX+ 395, 16c/32t, 64GB RAM |
| exo-0 | k3s worker | Framework Desktop |
| Namespace | Purpose |
|---|---|
argo |
Argo Workflows |
argocd |
ArgoCD controller |
buildgrid |
BuildGrid remote execution (controller, bots, CAS front, Postgres, one worker per node) |
buildbarn |
Buildbarn CAS, action cache, and remote asset (storage only) |
llm-d |
Local LLM inference (llama.cpp on Vulkan) |
local-registry |
Zot writable registry (30500) + pull-through cache (30501) |
| Application | Syncs path | Namespace |
|---|---|---|
testing-lab |
argo/workflow-templates/ |
argo |
testing-lab-infra |
manifests/ |
argo (+ others) |
- Edit
argo/workflow-templates/ormanifests/→ push tomain→ ArgoCD reconciles. - Never
kubectl applyorargo create workflow-templatefor tracked resources — ArgoCD overwrites them. argo/bootstrap/is not synced — run once by hand during cluster setup.
See docs/ops/bootstrap.md for full setup.
just setup-argocd # once
just argocd-sync
just run-bst-build # dakota BST build on BuildGrid
just run-dakota-qa # Dakota QA
just run-bluefin-server-build # bluefin-server BST build
just run-zot-promotion ... # promote a Zot candidate
just list-workflowsNo persistent VMs — boot-test VMs are ephemeral and destroyed via onExit;
just list-vms shows zero VMs when no boot test is running.
API-only operator model — cluster reads and mutations go through the
just wrappers or argo/kubectl. No SSH to cluster nodes.
One administration pane — KubeStellar Console. No Grafana or parallel dashboard framework.
WorkflowTemplate over inline DAG — reusable pipeline logic lives in
argo/workflow-templates/; ArgoCD owns the template lifecycle.
| Doc | Purpose |
|---|---|
| AGENTS.md | Agent entry point |
| docs/reference/workflow-reference.md | WorkflowTemplate submit interface and reference |
| docs/ops/bootstrap.md | Replicate this lab from scratch |
| docs/ops/RUNBOOK.md | Architecture + failure modes |
| docs/reference/agent-cheatsheet.md | Command reference |
| docs/ops/lab-operations.md | Operator procedures |