Guide for building custom node images

README.md

@@ -122,3 +122,7 @@ bink cluster stop --remove-data
 ```bash
 source <(bink completion bash)
 ```
+
+## Documentation
+
+- [Building Custom Node Images](docs/custom-node-images.md) — How to build and test with custom or patched packages

docs/custom-node-images.md

@@ -0,0 +1,337 @@
+# Building Custom Node Images
+
+## Overview
+
+When debugging a bug in a node component (bootc, CRI-O, kubelet, kernel, etc.), you need to replace the package or binary with a custom build and verify the fix. There are two approaches:
+
+1. **Runtime update** — Build a custom bootc OCI image and use `bootc switch` to deploy it on a running node. Faster, no disk rebuild needed.
+2. **Disk image rebuild** — Rebuild the entire qcow2 disk image. Required when the fix must be present from first boot (kernel, bootc itself, initrd).
+
+**When to use which:**
+
+| Situation | Approach |
+|-----------|----------|
+| Patching a userspace component (CRI-O, kubelet, a CLI tool) | Runtime update |
+| Patching bootc itself, the kernel, or initrd | Disk image rebuild |
+| Need the fix before the first boot completes | Disk image rebuild |
+
+## Prerequisites
+
+- Podman with socket running (`systemctl --user start podman.socket`)
+- `/dev/kvm` and `/dev/fuse` accessible
+- bink binary built (see [README](../README.md))
+- A running bink cluster, or you will create one
+
+## Inspecting the Current Node State
+
+Before making changes, inspect what is currently deployed. SSH into a node:
+
+```bash
+bink node ssh node1 --cluster-name <name>
+```
+
+Then run:
+
+```bash
+# Check the ostree deployment
+ostree admin status
+
+# Show ostree commit details
+ostree show --repo=/sysroot/ostree/repo <checksum>
+
+# Check the origin file (which container image the node tracks)
+sudo cat /sysroot/ostree/deploy/default/deploy/*.origin
+
+# Full bootc status
+sudo bootc status --json | jq
+
+# Check specific package versions
+rpm -qa | grep -E 'bootc|cri-o|kubeadm'
+```
+
+## Approach 1: Runtime Update via `bootc switch`
+
+The simpler path when the patch does not need to be in the boot image. Build a custom bootc OCI image, push it to bink's local registry, and switch the node to it.
+
+### Example: Building bootc from Source
+
+The [bootc repository](https://github.com/bootc-dev/bootc) has a Dockerfile with `ARG base=quay.io/centos-bootc/centos-bootc:stream10`. Override this to use the bink node image as the base, producing a new bootable container image with bootc compiled from any git commit:
+
+```bash
+git clone https://github.com/bootc-dev/bootc.git
+cd bootc
+git checkout <commit-or-branch>
+
+podman build \
+  --build-arg base=ghcr.io/bootc-dev/bink/node:v1.35-fedora-44 \
+  -t localhost/custom-bootc-node:latest \
+  .
+```
+
+This compiles bootc from source and installs it into an image derived from the bink node image, preserving all the existing packages (kubernetes, CRI-O, etc.).
+
+### Example: Layering a Custom Package
+
+For simpler cases (installing a different package version), write a minimal Containerfile:
+
+```dockerfile
+FROM ghcr.io/bootc-dev/bink/node:v1.35-fedora-44
+RUN dnf -y install <your-package>
+```
+
+Build it:
+
+```bash
+podman build -t localhost/custom-node:latest -f Containerfile.custom .
+```
+
+### Push to the Local Registry and Switch
+
+Bink runs a local OCI registry on port 5000. From the host, push to `localhost:5000`. Inside the VMs, the registry is reachable at `registry.cluster.local:5000`.
+
+```bash
+podman push --tls-verify=false \
+  localhost/custom-bootc-node:latest \
+  localhost:5000/custom-bootc-node:latest
+```
+
+SSH into the node and switch:
+
+```bash
+bink node ssh node1 --cluster-name <name>
+
+sudo bootc switch \
+  registry.cluster.local:5000/custom-bootc-node:latest
+```
+
+To switch by digest:
+
+```bash
+sudo bootc switch \
+  registry.cluster.local:5000/custom-bootc-node@sha256:<digest>
+```
+
+Reboot to apply:
+
+```bash
+sudo reboot
+```
+
+### Verify After Reboot
+
+```bash
+bink node ssh node1 --cluster-name <name>
+
+ostree admin status
+sudo bootc status --json | jq '.status.booted.image'
+rpm -q bootc
+```
+
+### Using `--target-imgref` for New Clusters
+
+Instead of switching after boot, set the tracked image reference at cluster creation:
+
+```bash
+bink cluster start \
+  --cluster-name test \
+  --target-imgref registry.cluster.local:5000/custom-bootc-node:latest
+```
+
+This rewrites the ostree origin file during cloud-init so the node tracks your custom image from the start. Running `bootc upgrade` will pull updates from your custom image reference.
+
+## Approach 2: Rebuild the Disk Image
+
+Required when the fix must be present from first boot (kernel, bootc, initrd changes).
+
+### Build Pipeline Overview
+
+The node disk image is built in two stages from `node-images/fedora/`:
+
+```
+Containerfile       ->  bootc OCI image   (bootc-base-imagectl build-rootfs)
+Containerfile.disk  ->  qcow2 disk image  (bcvk to-disk)
+```
+
+- **Stage 1** (`Containerfile`): Builds a bootc OCI image from `quay.io/fedora/fedora-bootc:44`. Uses `bootc-base-imagectl build-rootfs` with `--install` flags for packages (kubernetes, CRI-O, etc.). Validated with `bootc container lint`.
+- **Stage 2** (`Containerfile.disk`): Converts the bootc OCI image to a qcow2 disk using `bcvk to-disk`. The final container image contains only `/disk.qcow2` and `/images.txt`.
+
+> **Important:** Do not run `podman build` directly on `node-images/fedora/Containerfile`. The `build-rootfs` command requires elevated privileges. Always use the Makefile targets (`make build-bootc-image`, `make build-disk-image`) which pass the correct flags (`--cap-add=all --security-opt=label=disable --device /dev/fuse`).
+
+### Scenario A: Custom Package from a COPR Repo
+
+Modify `node-images/fedora/Containerfile` to add a COPR repo before the `build-rootfs` command:
+
+```dockerfile
+RUN dnf -y install 'dnf5-command(copr)' && \
+    dnf -y copr enable <owner>/<project> fedora-44-x86_64
+```
+
+The COPR architecture string must match the base image (e.g., `fedora-44-x86_64` for Fedora 44).
+
+Build and test:
+
+```bash
+cd node-images/fedora
+
+make build-bootc-image BOOTC_IMAGE=localhost/custom-node:latest
+make build-disk-image \
+  BOOTC_IMAGE=localhost/custom-node:latest \
+  NODE_IMAGE=localhost/custom-node:disk
+
+cd ../..
+
+bink cluster start \
+  --node-image localhost/custom-node:disk \
+  --cluster-name custom-test
+```
+
+> **Note:** COPR repos should be used as an opt-in customization for debugging and testing, not as a permanent change to the default image. Upstream COPR updates can introduce unexpected changes that break CI.
+
+### Scenario B: Local RPM File
+
+Copy the RPM into the build context and install it into the target rootfs after `build-rootfs` completes:
+
+```dockerfile
+COPY my-package.rpm /tmp/my-package.rpm
+
+# After the build-rootfs RUN instruction:
+RUN dnf --installroot=/target-rootfs install -y /tmp/my-package.rpm
+```
+
+The `--installroot=/target-rootfs` flag is required because `build-rootfs` assembles the filesystem at `/target-rootfs`, not in the builder's own root.
+
+Build and test with the same `make` commands as Scenario A.
+
+### Scenario C: Custom Binary Replacement
+
+Use a multi-stage build to compile the binary and copy it into the node image:
+
+```dockerfile
+FROM fedora:44 AS custom-build
+RUN dnf -y install git golang make
+COPY my-source/ /src
+WORKDIR /src
+RUN make build
+
+FROM quay.io/fedora/fedora-bootc:44 AS builder
+ARG KUBE_MINOR=1.35
+ARG KERNEL_VERSION=6.19.14-300.fc44
+RUN /usr/libexec/bootc-base-imagectl build-rootfs \
+    --manifest=minimal \
+    --no-docs \
+    --lock kernel-${KERNEL_VERSION} \
+    --lock kernel-core-${KERNEL_VERSION} \
+    --lock kernel-modules-${KERNEL_VERSION} \
+    --lock kernel-modules-core-${KERNEL_VERSION} \
+    --install NetworkManager \
+    --install openssh-server \
+    --install openssh-clients \
+    --install kubernetes \
+    --install kubernetes${KUBE_MINOR}-kubeadm \
+    --install kubernetes${KUBE_MINOR}-client \
+    --install cri-o${KUBE_MINOR} \
+    --install qemu-guest-agent \
+    --install bind-utils \
+    --install iputils \
+    --install cloud-init \
+    --install dnsmasq \
+    --install bubblewrap \
+    --install sudo \
+    --install vim-minimal \
+    --install jq \
+    --install less \
+    /target-rootfs
+
+FROM scratch AS root
+COPY --from=builder /target-rootfs/ /
+COPY --from=custom-build /src/my-binary /usr/bin/my-binary
+
+RUN passwd -d root
+RUN sed -i 's|"/opt/cni/bin"|"/var/lib/cni/bin"|g' /etc/crio/crio.conf && \
+    sed -i 's|"/opt/cni/net.d"|"/etc/cni/net.d"|g' /etc/crio/crio.conf
+RUN bootc container lint
+
+LABEL containers.bootc 1
+LABEL ostree.bootable 1
+STOPSIGNAL SIGRTMIN+3
+CMD ["/sbin/init"]
+```
+
+Build and test with the same `make` commands as Scenario A.
+
+### Composefs Variant
+
+To build the disk image with the composefs backend:
+
+```bash
+make build-disk-image-composefs \
+  BOOTC_IMAGE=localhost/custom-node:latest \
+  NODE_IMAGE=localhost/custom-node:disk-composefs
+```
+
+## Testing with Multiple Nodes
+
+Use two nodes to compare stock vs. custom behavior:
+
+```bash
+# Start a cluster with the stock image
+bink cluster start --cluster-name test
+
+# Add a worker with the custom disk image
+bink node add node2 --cluster-name test \
+  --node-image localhost/custom-node:disk
+
+# SSH into each to compare
+bink node ssh node1 --cluster-name test
+bink node ssh node2 --cluster-name test
+```
+
+Alternatively, switch a single worker at runtime:
+
+```bash
+bink node add node2 --cluster-name test
+
+# SSH into node2 and switch to the custom image
+bink node ssh node2 --cluster-name test
+sudo bootc switch \
+  registry.cluster.local:5000/custom-node:latest
+sudo reboot
+```
+
+## Makefile Variable Reference
+
+Variables in `node-images/fedora/Makefile`:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `KUBE_MINOR` | `1.35` | Kubernetes minor version |
+| `FEDORA_VERSION` | `44` | Fedora base version |
+| `DISK_SIZE` | `10G` | VM disk size |
+| `BUILD_MEMORY` | `4G` | Memory for bcvk build |
+| `BOOTC_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44` | Bootc OCI image name |
+| `NODE_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk` | Disk image name |
+| `BCVK_EXTRA_ARGS` | (none) | Extra flags for `bcvk to-disk` |
+
+## Troubleshooting
+
+### KVM Permission Errors
+
+```
+qemu-system-x86_64: Could not access KVM kernel module: Permission denied
+```
+
+Add your user to the `kvm` group or set permissions: `sudo chmod 666 /dev/kvm`.
+
+### `bootc container lint` Failures
+
+The Containerfile runs `bootc container lint` at the end. Custom modifications must preserve:
+- `/sbin/init` as the entrypoint (`CMD ["/sbin/init"]`)
+- Labels: `containers.bootc 1` and `ostree.bootable 1`
+
+### Disk Space
+
+The qcow2 build via bcvk needs approximately 15 GB of temporary space. Free disk space or reduce the disk size with `DISK_SIZE=8G`.
+
+### COPR Repo Not Found
+
+Verify the Fedora version and architecture string matches the base image. For `quay.io/fedora/fedora-bootc:44`, use `fedora-44-x86_64`.