From f6363cee2573d58acd7be1bbd66a1c546825b0d1 Mon Sep 17 00:00:00 2001 From: HarshwardhanPatil07 Date: Sat, 11 Jul 2026 15:30:51 +0530 Subject: [PATCH] docs: add guide for building custom node images Document how to build a disk image with a custom or patched package and test it with bink. Covers two approaches: - Runtime update via bootc switch (no disk rebuild) - Full disk image rebuild for boot-critical changes Includes concrete examples for COPR repos, local RPMs, binary replacement, and building bootc from source using the bootc repo's Dockerfile with the bink node image as base. Closes: #91 Signed-off-by: HarshwardhanPatil07 --- README.md | 4 + docs/custom-node-images.md | 337 +++++++++++++++++++++++++++++++++++++ 2 files changed, 341 insertions(+) create mode 100644 docs/custom-node-images.md diff --git a/README.md b/README.md index 6186ba8..7a82976 100644 --- a/README.md +++ b/README.md @@ -122,3 +122,7 @@ bink cluster stop --remove-data ```bash source <(bink completion bash) ``` + +## Documentation + +- [Building Custom Node Images](docs/custom-node-images.md) — How to build and test with custom or patched packages diff --git a/docs/custom-node-images.md b/docs/custom-node-images.md new file mode 100644 index 0000000..f0d47c8 --- /dev/null +++ b/docs/custom-node-images.md @@ -0,0 +1,337 @@ +# Building Custom Node Images + +## Overview + +When debugging a bug in a node component (bootc, CRI-O, kubelet, kernel, etc.), you need to replace the package or binary with a custom build and verify the fix. There are two approaches: + +1. **Runtime update** — Build a custom bootc OCI image and use `bootc switch` to deploy it on a running node. Faster, no disk rebuild needed. +2. **Disk image rebuild** — Rebuild the entire qcow2 disk image. Required when the fix must be present from first boot (kernel, bootc itself, initrd). + +**When to use which:** + +| Situation | Approach | +|-----------|----------| +| Patching a userspace component (CRI-O, kubelet, a CLI tool) | Runtime update | +| Patching bootc itself, the kernel, or initrd | Disk image rebuild | +| Need the fix before the first boot completes | Disk image rebuild | + +## Prerequisites + +- Podman with socket running (`systemctl --user start podman.socket`) +- `/dev/kvm` and `/dev/fuse` accessible +- bink binary built (see [README](../README.md)) +- A running bink cluster, or you will create one + +## Inspecting the Current Node State + +Before making changes, inspect what is currently deployed. SSH into a node: + +```bash +bink node ssh node1 --cluster-name +``` + +Then run: + +```bash +# Check the ostree deployment +ostree admin status + +# Show ostree commit details +ostree show --repo=/sysroot/ostree/repo + +# Check the origin file (which container image the node tracks) +sudo cat /sysroot/ostree/deploy/default/deploy/*.origin + +# Full bootc status +sudo bootc status --json | jq + +# Check specific package versions +rpm -qa | grep -E 'bootc|cri-o|kubeadm' +``` + +## Approach 1: Runtime Update via `bootc switch` + +The simpler path when the patch does not need to be in the boot image. Build a custom bootc OCI image, push it to bink's local registry, and switch the node to it. + +### Example: Building bootc from Source + +The [bootc repository](https://github.com/bootc-dev/bootc) has a Dockerfile with `ARG base=quay.io/centos-bootc/centos-bootc:stream10`. Override this to use the bink node image as the base, producing a new bootable container image with bootc compiled from any git commit: + +```bash +git clone https://github.com/bootc-dev/bootc.git +cd bootc +git checkout + +podman build \ + --build-arg base=ghcr.io/bootc-dev/bink/node:v1.35-fedora-44 \ + -t localhost/custom-bootc-node:latest \ + . +``` + +This compiles bootc from source and installs it into an image derived from the bink node image, preserving all the existing packages (kubernetes, CRI-O, etc.). + +### Example: Layering a Custom Package + +For simpler cases (installing a different package version), write a minimal Containerfile: + +```dockerfile +FROM ghcr.io/bootc-dev/bink/node:v1.35-fedora-44 +RUN dnf -y install +``` + +Build it: + +```bash +podman build -t localhost/custom-node:latest -f Containerfile.custom . +``` + +### Push to the Local Registry and Switch + +Bink runs a local OCI registry on port 5000. From the host, push to `localhost:5000`. Inside the VMs, the registry is reachable at `registry.cluster.local:5000`. + +```bash +podman push --tls-verify=false \ + localhost/custom-bootc-node:latest \ + localhost:5000/custom-bootc-node:latest +``` + +SSH into the node and switch: + +```bash +bink node ssh node1 --cluster-name + +sudo bootc switch \ + registry.cluster.local:5000/custom-bootc-node:latest +``` + +To switch by digest: + +```bash +sudo bootc switch \ + registry.cluster.local:5000/custom-bootc-node@sha256: +``` + +Reboot to apply: + +```bash +sudo reboot +``` + +### Verify After Reboot + +```bash +bink node ssh node1 --cluster-name + +ostree admin status +sudo bootc status --json | jq '.status.booted.image' +rpm -q bootc +``` + +### Using `--target-imgref` for New Clusters + +Instead of switching after boot, set the tracked image reference at cluster creation: + +```bash +bink cluster start \ + --cluster-name test \ + --target-imgref registry.cluster.local:5000/custom-bootc-node:latest +``` + +This rewrites the ostree origin file during cloud-init so the node tracks your custom image from the start. Running `bootc upgrade` will pull updates from your custom image reference. + +## Approach 2: Rebuild the Disk Image + +Required when the fix must be present from first boot (kernel, bootc, initrd changes). + +### Build Pipeline Overview + +The node disk image is built in two stages from `node-images/fedora/`: + +``` +Containerfile -> bootc OCI image (bootc-base-imagectl build-rootfs) +Containerfile.disk -> qcow2 disk image (bcvk to-disk) +``` + +- **Stage 1** (`Containerfile`): Builds a bootc OCI image from `quay.io/fedora/fedora-bootc:44`. Uses `bootc-base-imagectl build-rootfs` with `--install` flags for packages (kubernetes, CRI-O, etc.). Validated with `bootc container lint`. +- **Stage 2** (`Containerfile.disk`): Converts the bootc OCI image to a qcow2 disk using `bcvk to-disk`. The final container image contains only `/disk.qcow2` and `/images.txt`. + +> **Important:** Do not run `podman build` directly on `node-images/fedora/Containerfile`. The `build-rootfs` command requires elevated privileges. Always use the Makefile targets (`make build-bootc-image`, `make build-disk-image`) which pass the correct flags (`--cap-add=all --security-opt=label=disable --device /dev/fuse`). + +### Scenario A: Custom Package from a COPR Repo + +Modify `node-images/fedora/Containerfile` to add a COPR repo before the `build-rootfs` command: + +```dockerfile +RUN dnf -y install 'dnf5-command(copr)' && \ + dnf -y copr enable / fedora-44-x86_64 +``` + +The COPR architecture string must match the base image (e.g., `fedora-44-x86_64` for Fedora 44). + +Build and test: + +```bash +cd node-images/fedora + +make build-bootc-image BOOTC_IMAGE=localhost/custom-node:latest +make build-disk-image \ + BOOTC_IMAGE=localhost/custom-node:latest \ + NODE_IMAGE=localhost/custom-node:disk + +cd ../.. + +bink cluster start \ + --node-image localhost/custom-node:disk \ + --cluster-name custom-test +``` + +> **Note:** COPR repos should be used as an opt-in customization for debugging and testing, not as a permanent change to the default image. Upstream COPR updates can introduce unexpected changes that break CI. + +### Scenario B: Local RPM File + +Copy the RPM into the build context and install it into the target rootfs after `build-rootfs` completes: + +```dockerfile +COPY my-package.rpm /tmp/my-package.rpm + +# After the build-rootfs RUN instruction: +RUN dnf --installroot=/target-rootfs install -y /tmp/my-package.rpm +``` + +The `--installroot=/target-rootfs` flag is required because `build-rootfs` assembles the filesystem at `/target-rootfs`, not in the builder's own root. + +Build and test with the same `make` commands as Scenario A. + +### Scenario C: Custom Binary Replacement + +Use a multi-stage build to compile the binary and copy it into the node image: + +```dockerfile +FROM fedora:44 AS custom-build +RUN dnf -y install git golang make +COPY my-source/ /src +WORKDIR /src +RUN make build + +FROM quay.io/fedora/fedora-bootc:44 AS builder +ARG KUBE_MINOR=1.35 +ARG KERNEL_VERSION=6.19.14-300.fc44 +RUN /usr/libexec/bootc-base-imagectl build-rootfs \ + --manifest=minimal \ + --no-docs \ + --lock kernel-${KERNEL_VERSION} \ + --lock kernel-core-${KERNEL_VERSION} \ + --lock kernel-modules-${KERNEL_VERSION} \ + --lock kernel-modules-core-${KERNEL_VERSION} \ + --install NetworkManager \ + --install openssh-server \ + --install openssh-clients \ + --install kubernetes \ + --install kubernetes${KUBE_MINOR}-kubeadm \ + --install kubernetes${KUBE_MINOR}-client \ + --install cri-o${KUBE_MINOR} \ + --install qemu-guest-agent \ + --install bind-utils \ + --install iputils \ + --install cloud-init \ + --install dnsmasq \ + --install bubblewrap \ + --install sudo \ + --install vim-minimal \ + --install jq \ + --install less \ + /target-rootfs + +FROM scratch AS root +COPY --from=builder /target-rootfs/ / +COPY --from=custom-build /src/my-binary /usr/bin/my-binary + +RUN passwd -d root +RUN sed -i 's|"/opt/cni/bin"|"/var/lib/cni/bin"|g' /etc/crio/crio.conf && \ + sed -i 's|"/opt/cni/net.d"|"/etc/cni/net.d"|g' /etc/crio/crio.conf +RUN bootc container lint + +LABEL containers.bootc 1 +LABEL ostree.bootable 1 +STOPSIGNAL SIGRTMIN+3 +CMD ["/sbin/init"] +``` + +Build and test with the same `make` commands as Scenario A. + +### Composefs Variant + +To build the disk image with the composefs backend: + +```bash +make build-disk-image-composefs \ + BOOTC_IMAGE=localhost/custom-node:latest \ + NODE_IMAGE=localhost/custom-node:disk-composefs +``` + +## Testing with Multiple Nodes + +Use two nodes to compare stock vs. custom behavior: + +```bash +# Start a cluster with the stock image +bink cluster start --cluster-name test + +# Add a worker with the custom disk image +bink node add node2 --cluster-name test \ + --node-image localhost/custom-node:disk + +# SSH into each to compare +bink node ssh node1 --cluster-name test +bink node ssh node2 --cluster-name test +``` + +Alternatively, switch a single worker at runtime: + +```bash +bink node add node2 --cluster-name test + +# SSH into node2 and switch to the custom image +bink node ssh node2 --cluster-name test +sudo bootc switch \ + registry.cluster.local:5000/custom-node:latest +sudo reboot +``` + +## Makefile Variable Reference + +Variables in `node-images/fedora/Makefile`: + +| Variable | Default | Description | +|----------|---------|-------------| +| `KUBE_MINOR` | `1.35` | Kubernetes minor version | +| `FEDORA_VERSION` | `44` | Fedora base version | +| `DISK_SIZE` | `10G` | VM disk size | +| `BUILD_MEMORY` | `4G` | Memory for bcvk build | +| `BOOTC_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44` | Bootc OCI image name | +| `NODE_IMAGE` | `ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk` | Disk image name | +| `BCVK_EXTRA_ARGS` | (none) | Extra flags for `bcvk to-disk` | + +## Troubleshooting + +### KVM Permission Errors + +``` +qemu-system-x86_64: Could not access KVM kernel module: Permission denied +``` + +Add your user to the `kvm` group or set permissions: `sudo chmod 666 /dev/kvm`. + +### `bootc container lint` Failures + +The Containerfile runs `bootc container lint` at the end. Custom modifications must preserve: +- `/sbin/init` as the entrypoint (`CMD ["/sbin/init"]`) +- Labels: `containers.bootc 1` and `ostree.bootable 1` + +### Disk Space + +The qcow2 build via bcvk needs approximately 15 GB of temporary space. Free disk space or reduce the disk size with `DISK_SIZE=8G`. + +### COPR Repo Not Found + +Verify the Fedora version and architecture string matches the base image. For `quay.io/fedora/fedora-bootc:44`, use `fedora-44-x86_64`.