Let a provider authenticate its discovery fetches with a client certificate + kubelet auth provider

.github/workflows/build.yaml

@@ -593,9 +593,11 @@ jobs:
           - os: ubuntu-latest
             target: x86_64-unknown-linux-musl
             binary_name: angos-linux-amd64
+            provider_binary_name: angos-credential-provider-linux-amd64
           - os: ubuntu-24.04-arm
             target: aarch64-unknown-linux-musl
             binary_name: angos-linux-arm64
+            provider_binary_name: angos-credential-provider-linux-arm64
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v7
@@ -630,6 +632,26 @@ jobs:
           if-no-files-found: error
           retention-days: 1
 
+      # The kubelet credential provider is installed on nodes, it ships for
+      # Linux alone.
+      - name: Build kubelet credential provider
+        if: contains(matrix.target, 'linux')
+        env:
+          CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_RUSTFLAGS: "-Clink-self-contained=yes -Clinker=rust-lld"
+          CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS: "-Clink-self-contained=yes -Clinker=rust-lld"
+        run: |
+          cargo build --release --target ${{ matrix.target }} -p kubelet-credential-provider
+          mv target/${{ matrix.target }}/release/angos-credential-provider ${{ matrix.provider_binary_name }}
+
+      - name: Upload kubelet credential provider
+        if: contains(matrix.target, 'linux')
+        uses: actions/upload-artifact@v7
+        with:
+          name: binary-${{ matrix.provider_binary_name }}
+          path: ${{ matrix.provider_binary_name }}
+          if-no-files-found: error
+          retention-days: 1
+
   build-container:
     runs-on: ubuntu-latest
     strategy:
@@ -793,6 +815,17 @@ jobs:
             | Linux amd64          | `angos-linux-amd64`    |
             | Linux arm64          | `angos-linux-arm64`    |
 
+            ## Kubelet Credential Provider
+
+            Installed on nodes so image pulls authenticate with the pulling pod's
+            service-account token. `contrib/kubelet-credential-provider` carries a
+            DaemonSet that installs it from this release.
+
+            | Platform    | Binary                                    |
+            |-------------|-------------------------------------------|
+            | Linux amd64 | `angos-credential-provider-linux-amd64`   |
+            | Linux arm64 | `angos-credential-provider-linux-arm64`   |
+
             ## Verification with Cosign
 
             All release artifacts are signed with [Sigstore Cosign](https://docs.sigstore.dev/)

CHANGELOG.md

@@ -11,6 +11,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 - An optional token service exchanges a client's credential for a registry-signed bearer token at `GET /token`, so a short-lived CI credential no longer has to outlive the push it starts.
 - `auth.oidc.<name>.required_claims` rejects a token that does not carry the claims listed, before any access policy runs.
 - `auth.oidc.<name>.server_ca_bundle` trusts a private CA for that provider's discovery and JWKS fetches, so an issuer such as a kube-apiserver needs no host-wide trust.
+- `auth.oidc.<name>.client_certificate_bundle` and `client_private_key` present a client certificate on those fetches, so a cluster that keeps discovery closed to unauthenticated users can still back image pulls with projected service-account tokens.
+- EXPERIMENTAL: `contrib/kubelet-credential-provider` hands the kubelet the pulling pod's service-account token as its registry credential, so an image pull authenticates as the workload instead of a shared `imagePullSecret`. It ships as a released Linux binary, with a DaemonSet that installs it on every node and restarts the kubelet only when the binary changes.
 - The `has_repository_policy()` access-policy function lets a global rule admit only what a `[repository]` declaring its own `access_policy` will decide, instead of restating every repository rule globally.
 
 ### Changed

Cargo.toml

@@ -1,5 +1,6 @@
 [workspace]
 members = [
+    "contrib/kubelet-credential-provider",
     "crates/backoff",
     "crates/conformance-gates",
     "crates/s3-client",

Dockerfile

@@ -24,6 +24,7 @@
 
 COPY Cargo.toml Cargo.lock build.rs ./
 COPY crates ./crates
+COPY contrib/kubelet-credential-provider ./contrib/kubelet-credential-provider
 COPY src ./src
 COPY ui ./ui
 
@@ -38,7 +39,7 @@
     cargo build --target=$TOOLCHAIN $BUILD_FLAG; \
     mv "target/$TOOLCHAIN/$RELEASE_MODE/angos" target/angos
 
 FROM --platform=$TARGETPLATFORM scratch AS final
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
 COPY --from=build /buildroot/target/angos /angos
 EXPOSE 8000

contrib/kubelet-credential-provider/Cargo.toml

@@ -0,0 +1,17 @@
+[package]
+name = "kubelet-credential-provider"
+version = "0.1.0"
+edition = "2024"
+
+[lints.clippy]
+pedantic = { level = "warn", priority = -1 }
+
+[[bin]]
+name = "angos-credential-provider"
+path = "src/main.rs"
+
+[dependencies]
+argh = { workspace = true }
+base64 = { workspace = true }
+serde = { workspace = true }
+serde_json = { workspace = true }

contrib/kubelet-credential-provider/README.md

@@ -0,0 +1,47 @@
+# angos kubelet credential provider
+
+Lets the kubelet pull images with the service-account token of the pod it is
+starting, instead of a static `imagePullSecret`.
+
+The kubelet mints the token, execs this plugin with it on stdin, and gets back
+the registry credential angos accepts: the `[auth.oidc.<name>]` section name as
+the username, the token as the password.
+
+It builds under the workspace release profile the registry itself uses, so
+`cargo build --release -p kubelet-credential-provider` already applies fat LTO
+and `opt-level = 3`; add `--target x86_64-unknown-linux-musl` for the static
+binary the nodes want. Install and configure it as described in
+[Configure Kubernetes OIDC](../../doc/how-to/configure-kubernetes-oidc.md#pulling-images-with-the-kubelet).
+
+## Installing on every node
+
+`daemonset.yaml` installs the released binary on each node and restarts the
+kubelet, which resolves its configured plugins once at startup and so ignores a
+binary that appeared later. Set `VERSION` and both checksums first:
+
+```bash
+VERSION=v1.5.0
+for arch in amd64 arm64; do
+  curl -sL "https://github.com/project-angos/angos/releases/download/${VERSION}/angos-credential-provider-linux-${arch}" \
+    | sha256sum
+done
+kubectl apply -f daemonset.yaml
+```
+
+Re-running costs one checksum: the installer compares what the node already has
+against the pinned value and exits without downloading or restarting anything
+when they match. A download that fails verification leaves the node's existing
+binary in place, and the kubelet is restarted only after a new binary lands.
+
+The installer does not touch kubelet flags. Point the kubelet at the plugin with
+`--image-credential-provider-config` and `--image-credential-provider-bin-dir`
+through whatever bootstraps your nodes: rewriting a unit file from a DaemonSet
+would fight the distribution that owns it. Note also that a first rollout
+restarts the kubelet on every node at once, so apply it to a `nodeSelector`
+subset first if that matters to you.
+
+The exchange follows `k8s.io/kubelet/pkg/apis/credentialprovider/v1`, and the
+`tokenAttributes` that make the kubelet mint the token are defined by
+`CredentialProvider` in `k8s.io/kubelet/config/v1`. Both are published from
+`staging/src/k8s.io/kubelet` in kubernetes/kubernetes and rendered at
+[kubernetes.io/docs/reference/config-api/kubelet-credentialprovider.v1](https://kubernetes.io/docs/reference/config-api/kubelet-credentialprovider.v1/).

contrib/kubelet-credential-provider/daemonset.yaml

@@ -0,0 +1,132 @@
+# Installs the angos kubelet credential provider on every node from a GitHub
+# release, then restarts the kubelet so it picks the plugin up.
+#
+# Idempotent: the installer compares the checksum already on the node with the
+# one configured here and exits without downloading or restarting anything when
+# they match, so a rescheduled pod or a node reboot costs one checksum.
+#
+# Set VERSION and both checksums before applying. The kubelet flags
+# (--image-credential-provider-config, --image-credential-provider-bin-dir) stay
+# a node-bootstrap concern; this only keeps the binary current.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: angos-credential-provider-installer
+  namespace: kube-system
+data:
+  install.sh: |
+    #!/bin/sh
+    set -eu
+
+    case "$(uname -m)" in
+      x86_64)  asset="angos-credential-provider-linux-amd64"; expected="$SHA256_AMD64" ;;
+      aarch64) asset="angos-credential-provider-linux-arm64"; expected="$SHA256_ARM64" ;;
+      *) echo "unsupported architecture $(uname -m)" >&2; exit 1 ;;
+    esac
+
+    if [ -z "$expected" ]; then
+      echo "no checksum configured for $asset; refusing to install an unverified binary" >&2
+      exit 1
+    fi
+
+    target="/host${BIN_DIR}/angos-credential-provider"
+    installed=""
+    if [ -f "$target" ]; then
+      installed="$(sha256sum "$target" | cut -d' ' -f1)"
+    fi
+
+    if [ "$installed" = "$expected" ]; then
+      echo "$target is already ${VERSION}; leaving the kubelet alone"
+      exit 0
+    fi
+
+    work="$(mktemp -d)"
+    trap 'rm -rf "$work"' EXIT
+    wget -qO "$work/$asset" \
+      "https://github.com/project-angos/angos/releases/download/${VERSION}/${asset}"
+    echo "$expected  $asset" | (cd "$work" && sha256sum -c -)
+
+    # Install beside the target and rename: a pull racing the write never sees a
+    # half-written plugin.
+    mkdir -p "/host${BIN_DIR}"
+    install -m 0755 "$work/$asset" "$target.new"
+    mv "$target.new" "$target"
+
+    # The kubelet resolves the configured plugins once at startup, so a binary
+    # that appeared or changed since then is only used after a restart.
+    chroot /host systemctl restart kubelet
+    echo "installed ${VERSION} and restarted the kubelet"
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: angos-credential-provider-installer
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: angos-credential-provider
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: angos-credential-provider
+  updateStrategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 1
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: angos-credential-provider
+    spec:
+      priorityClassName: system-node-critical
+      automountServiceAccountToken: false
+      tolerations:
+        - operator: Exists
+      initContainers:
+        - name: install
+          image: alpine:3.22
+          command: ["/bin/sh", "/script/install.sh"]
+          env:
+            - name: VERSION
+              value: v1.5.0
+            - name: SHA256_AMD64
+              value: ""
+            - name: SHA256_ARM64
+              value: ""
+            - name: BIN_DIR
+              value: /var/lib/kubelet/credential-providers
+          securityContext:
+            # Writes into the kubelet's bin dir and restarts the host unit.
+            privileged: true
+          volumeMounts:
+            - name: host
+              mountPath: /host
+            - name: script
+              mountPath: /script
+              readOnly: true
+      containers:
+        # The install runs to completion in the init container; this only keeps
+        # the pod scheduled, so it holds no host access at all.
+        - name: idle
+          image: alpine:3.22
+          command: ["/bin/sh", "-c", "trap 'exit 0' TERM; sleep infinity & wait"]
+          resources:
+            requests:
+              cpu: 1m
+              memory: 8Mi
+            limits:
+              memory: 16Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            runAsNonRoot: true
+            runAsUser: 65534
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: host
+          hostPath:
+            path: /
+        - name: script
+          configMap:
+            name: angos-credential-provider-installer
+            defaultMode: 0755