diff --git a/docs/README.md b/docs/README.md index c8fe8de1c..ee1c4933c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -151,8 +151,8 @@ A skill link names a procedure; the entry's other fields carry shared facts. - ID: `development` - Use for: build, lint, test and codegen commands; commit and PR conventions; contributor invariants; release steps; site publication and translations. - Aliases: `make`, `lint`, `codegen`, `vendored dependencies`, `staging`, `Hugo`, `GitHub Pages` -- Guides: commands and targets in [Development](contribute/development.md); startup invariants in [Internals](contribute/internals.md); orientation in [Architecture](getting-started/architecture.md). -- Code: [Makefile](../Makefile) and the scripts behind it in [hack](../hack/) · site publication in [Site workflow](../.github/workflows/site.yml) and [publisher](../hack/site/publish.py) · patched modules in [staging](../staging/) · generators in [gen](../gen/) +- Guides: commands and targets in [Development](contribute/development.md); startup invariants in [Internals](contribute/internals.md); orientation in [Architecture](getting-started/architecture.md); the disposable local cluster recipe in [Local API Testing](contribute/local-api-testing.md). +- Code: [Makefile](../Makefile) and the scripts behind it in [hack](../hack/) · site publication in [Site workflow](../.github/workflows/site.yml) and [publisher](../hack/site/publish.py) · patched modules in [staging](../staging/) · generators in [gen](../gen/) · the local cluster recipe in [testing/infra/local_worker](../testing/infra/local_worker/) - Related: [API Changes](#api-changes) (what `make generate` owns) · [Installation](#installation) (chart targets and vendoring) - Skills: [gpustack-operator-overview](../.agents/skills/gpustack-operator-overview/SKILL.md) (codebase tour) · [gpustack-operator-docs](../.agents/skills/gpustack-operator-docs/SKILL.md) (documentation changes) · [gpustack-operator-code-review](../.agents/skills/gpustack-operator-code-review/SKILL.md) (PR review criteria) · [gpustack-operator-issue-pr](../.agents/skills/gpustack-operator-issue-pr/SKILL.md) (issue and PR titles) · [gpustack-operator-release](../.agents/skills/gpustack-operator-release/SKILL.md) (cut a release) · [gpustack-operator-generate](../.agents/skills/gpustack-operator-generate/SKILL.md) (API codegen) @@ -292,6 +292,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. |---|---| | [Development](contribute/development.md) | Build, generate and lint the project. | | [Internals](contribute/internals.md) | Review startup and naming constraints. | +| [Local API Testing](contribute/local-api-testing.md) | Create a disposable local cluster and probe the worker path. | ## Conventions diff --git a/docs/contribute/local-api-testing.md b/docs/contribute/local-api-testing.md new file mode 100644 index 000000000..9433710dc --- /dev/null +++ b/docs/contribute/local-api-testing.md @@ -0,0 +1,198 @@ +# Local API Testing + +A disposable kind cluster on the Mac proves that a control plane can reach a Mac-hosted worker. +The recipe creates one cluster, one kubeconfig and one node image, and removes only what it +created. + +## Contents + +- [Scope](#scope) +- [Prerequisites](#prerequisites) +- [Commands](#commands) +- [Cluster shape](#cluster-shape) +- [Network probe](#network-probe) +- [Isolation](#isolation) +- [Cleanup](#cleanup) +- [Tests](#tests) +- [Limitations](#limitations) + +## Scope + +The recipe proves three things. + +- A disposable cluster runs the pinned Kubernetes version on native arm64. +- The control plane enables aggregator routing and excludes the EndpointSlice mirroring controller. +- A concrete Mac address answers a plain HTTP request from inside the control-plane container. + +It does not prove worker aggregation. The authenticated worker, the aggregation request and the +serving certificate belong to a later stage of this upgrade, and a later stage adds the command +that exercises them to the same script. + +## Prerequisites + +- macOS on arm64. The recipe refuses any other host. +- Docker Engine serving `linux/arm64`. +- kind v0.33.0 or later. The build needs the `--type` and `--base-image` flags. +- `curl` and `python3` on the host. + +The recipe downloads its own kubectl and builds its own node image, so it needs network access to +`dl.k8s.io` and to the container registry. + +## Commands + +Run the commands in order from the repository root. + +```sh +bash testing/infra/local_worker/run.sh preflight +bash testing/infra/local_worker/run.sh prepare +bash testing/infra/local_worker/run.sh network +bash testing/infra/local_worker/run.sh cleanup +``` + +`preflight` checks the host, the tools, the cluster name and the node image situation, and it +creates nothing except the task-local kubectl. `prepare` builds or reuses the node image, creates +the cluster and proves its shape. `network` proves the reachability. `cleanup` removes the cluster +and the files. + +Every command accepts these environment variables. + +| Variable | Default | Purpose | +|---|---|---| +| `LOCAL_WORKER_CLUSTER_NAME` | `gpustack-local-worker` | The kind cluster name. | +| `LOCAL_WORKER_NAMESPACE` | `gpustack-local-worker-check` | The namespace the mirroring fixture is created in. | +| `LOCAL_WORKER_STATE_DIR` | `.dist/local_worker` | The kubeconfig, the kubectl and the ownership record. | +| `LOCAL_WORKER_NODE_IMAGE` | `gpustack-upgrade-node:v1.36.5-arm64` | The node image tag. | +| `LOCAL_WORKER_BASE_IMAGE` | `kindest/base:v20260820-69b56db7` | The base image the node image is built on. | +| `LOCAL_WORKER_RELEASE_TARBALL` | `.dist/local-worker-cache/kubernetes-server-linux-arm64.tar.gz` | The release tarball cache, outside the state directory. | + +The recipe pins the cluster and its kubectl to `v1.36.5`. `LOCAL_WORKER_KUBERNETES_VERSION` and +`LOCAL_WORKER_KUBECTL_VERSION` are those two pins. The base image, the release tarball and the check +that the cluster reports the pinned version name the same version, so a different value leaves a +cluster that no longer proves what this page documents. + +## Cluster shape + +The cluster is one control-plane node. The kubeadm arguments are name and value lists, the shape +the `kubeadm.k8s.io/v1beta4` API expects. + +```yaml +apiVersion: kubeadm.k8s.io/v1beta4 +kind: ClusterConfiguration +apiServer: + extraArgs: + - name: enable-aggregator-routing + value: "true" +controllerManager: + extraArgs: + - name: controllers + value: "*,-endpointslicemirroring" +``` + +The `*` prefix keeps every other controller. The exclusion token is the alias the controller +manager registers for the mirroring controller, and the recipe reads it back from the rendered +static Pod instead of trusting the configuration. + +| Setting | Value | Where the recipe reads it | +|---|---|---| +| Kubernetes version | `v1.36.5` | The version the server reports. | +| Node architecture | `arm64` | The node status. | +| Aggregator routing | `enable-aggregator-routing=true` | The `kube-apiserver` static Pod command and args. | +| EndpointSlice mirroring | `controllers=*,-endpointslicemirroring` | The `kube-controller-manager` static Pod command and args. | + +Both static Pods must also be running. A component that rejects an argument exits, and a +component that is not running mirrors nothing, so a missing slice would otherwise pass for the +wrong reason. + +The exclusion is then observed rather than assumed. The recipe creates a Service without a +selector and a core `Endpoints` object without the skip-mirror label, which are the conditions the +mirroring controller acts on, and waits a bounded window for a mirrored `EndpointSlice` that must +not appear. + +A second Service carries a selector, so the `EndpointSlice` controller owns its slices; its slice +has to appear in the same window. That control shows that `EndpointSlice` creation and +observation work in this cluster, so the absent slice is the excluded controller and not a dead +fixture. + +A read of the slices that fails ends the observation and fails the command. A failed read is never +counted as an absent slice, because the absence would rest on a read that never answered. + +## Network probe + +The probe is deliberately plain. The recipe resolves `host.docker.internal` inside the node, +binds a temporary listener on the Mac, and asks the control-plane container for a unique token +over plain HTTP. + +```sh +curl --fail --silent --show-error --max-time 15 http://
:/ +``` + +The listener serves a token generated for that run and answers requests until the command ends +or a two-minute deadline passes, so a stray connection cannot consume the probe. The recipe +records the address it used and removes the listener and its files when the command ends. + +No certificate, no subject alternative name and no identity are involved. The probe therefore +proves reachability only. A serving certificate that carries the address, an authenticated worker +and an aggregation request are the boundary a later stage owns. + +## Isolation + +Every kubectl and kind invocation names the task-local kubeconfig and the task-local context. The +recipe exports `KUBECONFIG` to the same path and refuses to run when that value points outside the +state directory. + +The default kubeconfig is never written. The recipe records its fingerprint when it creates the +cluster and compares it again afterwards. + +The cluster name is unique to the run. `prepare` refuses to continue when a cluster already holds +that name and the recipe has no record of creating it, so a foreign cluster is never adopted. + +The ownership record in the state directory holds the cluster name, the immutable identifier of +the node container, the kubeconfig path, the node image identifier and the fingerprint of the +default kubeconfig. + +## Cleanup + +`cleanup` removes the cluster only when the ownership record is complete, the record names the +cluster this run asked about, and the live node container carries the recorded identifier. A +record that is missing, damaged or names another cluster stops the removal instead of being read +as an absent record, and so does a live node container that does not match. + +The fingerprint of the default kubeconfig is compared again after the cluster is gone, before +the record that holds it is removed. + +The node image stays. It is a build cache, and a rebuild costs a full release download. + +The mirroring fixture lives in its own namespace inside that cluster. The recipe creates that +namespace itself and removes it when the proof ends, and also when an assertion fails, so a failed +proof leaves no fixture behind. A namespace that already existed is never adopted and never +deleted: the proof fails instead. + +## Tests + +```sh +bash testing/infra/local_worker/test_recipe.sh +``` + +The fast group calls the validation predicates with declarative inputs and needs no cluster. The +live group runs the recipe end to end, then proves that cleanup refuses a cluster it cannot prove +it created, a damaged ownership record and a record that names another cluster, and that a +repeated cleanup succeeds without changing anything. Set `LOCAL_WORKER_TEST_FAST_ONLY=true` to +run the fast group alone. + +## Limitations + +- The proof is network-only. No worker starts, no aggregation request is made and no serving + certificate is issued. +- The recipe runs on macOS arm64 only. +- The node image build needs the release tarball. kind downloads it with a deadline, and a slow + link can exceed it. Build from a checksum-verified local copy of the same tarball with + `--type file` when that happens; the resulting image is the same release. A cached copy whose + checksum no longer matches the published one is ignored, and the build downloads its own. + +--- + +**See also** — [Development](development.md) (build, lint and test commands) · +[Internals](internals.md) (the invariants the code keeps) · +[Scheduling Chain](../modules/devices/scheduling.md) + +**Next** → [All documentation](../README.md) — pick the next page on the contributor path. diff --git a/site/data/navigation.yaml b/site/data/navigation.yaml index a57ee42f5..71882dd81 100644 --- a/site/data/navigation.yaml +++ b/site/data/navigation.yaml @@ -103,6 +103,7 @@ order: /docs/contribute/: - development - internals + - local-api-testing overview_labels: /docs/walkthroughs/model-deployment/: Preparation and Choices children: @@ -188,3 +189,4 @@ labels: /docs/reference/instance-metrics/: Instance Metrics /docs/contribute/development/: Development /docs/contribute/internals/: Internals + /docs/contribute/local-api-testing/: Local API Testing diff --git a/testing/infra/local_worker/kind-config.yaml b/testing/infra/local_worker/kind-config.yaml new file mode 100644 index 000000000..076cc7ad8 --- /dev/null +++ b/testing/infra/local_worker/kind-config.yaml @@ -0,0 +1,16 @@ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + kubeadmConfigPatches: + - | + apiVersion: kubeadm.k8s.io/v1beta4 + kind: ClusterConfiguration + apiServer: + extraArgs: + - name: enable-aggregator-routing + value: "true" + controllerManager: + extraArgs: + - name: controllers + value: "*,-endpointslicemirroring" diff --git a/testing/infra/local_worker/lib.sh b/testing/infra/local_worker/lib.sh new file mode 100644 index 000000000..691252591 --- /dev/null +++ b/testing/infra/local_worker/lib.sh @@ -0,0 +1,573 @@ +#!/usr/bin/env bash +# +# Shared helpers for the disposable local cluster recipe. +# +# The functions split into two groups. The pure ones take their arguments and +# return a verdict, so the safety cases in test_recipe.sh can exercise them +# without a cluster. The rest reach Docker, kind and kubectl, and every one of +# them targets the task-owned resources named in the ownership record. + +set -o errexit +set -o nounset +set -o pipefail + +LW_REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd -P)" + +LW_KUBERNETES_VERSION="${LOCAL_WORKER_KUBERNETES_VERSION:-v1.36.5}" +LW_KUBECTL_VERSION="${LOCAL_WORKER_KUBECTL_VERSION:-v1.36.5}" +LW_NODE_IMAGE="${LOCAL_WORKER_NODE_IMAGE:-gpustack-upgrade-node:v1.36.5-arm64}" +LW_BASE_IMAGE="${LOCAL_WORKER_BASE_IMAGE:-kindest/base:v20260820-69b56db7}" +LW_CLUSTER_NAME="${LOCAL_WORKER_CLUSTER_NAME:-gpustack-local-worker}" +LW_NAMESPACE="${LOCAL_WORKER_NAMESPACE:-gpustack-local-worker-check}" + +LW_OWNERSHIP_KEYS="CLUSTER_NAME CONTEXT NODE_CONTAINER_NAME NODE_CONTAINER_ID KUBECONFIG KUBECTL NODE_IMAGE NODE_IMAGE_ID KUBERNETES_VERSION DEFAULT_KUBECONFIG_SHA256 CREATED_AT" + +# The outcomes of an EndpointSlice observation. A read that failed is an +# outcome of its own, so it is never read as an absence. +LW_SLICE_ABSENT=0 +LW_SLICE_PRESENT=1 +LW_SLICE_READ_FAILED=2 + +lw_log() { + printf '[local-worker] %s\n' "$*" +} + +lw_die() { + printf '[local-worker] error: %s\n' "$*" >&2 + exit 1 +} + +lw_require_cmd() { + local cmd + for cmd in "$@"; do + command -v "${cmd}" >/dev/null 2>&1 || lw_die "${cmd} is not on PATH" + done +} + +# The wait loops sleep through this helper, so a test can run them without +# waiting for the clock. +lw_sleep() { + sleep "$@" +} + +# Paths + +lw_state_dir() { + printf '%s\n' "${LOCAL_WORKER_STATE_DIR:-${LW_REPO_ROOT}/.dist/local_worker}" +} + +lw_state_file() { + printf '%s/ownership.env\n' "$(lw_state_dir)" +} + +lw_kubeconfig_path() { + printf '%s/kubeconfig\n' "$(lw_state_dir)" +} + +lw_kubectl_path() { + printf '%s/bin/kubectl\n' "$(lw_state_dir)" +} + +lw_context() { + printf 'kind-%s\n' "$LW_CLUSTER_NAME" +} + +lw_node_container_name() { + printf '%s-control-plane\n' "$LW_CLUSTER_NAME" +} + +lw_default_kubeconfig_path() { + printf '%s/.kube/config\n' "${HOME}" +} + +# Digests and fingerprints + +lw_sha256() { + if command -v shasum >/dev/null 2>&1; then + shasum -a 256 "$1" | awk '{print $1}' + elif command -v sha256sum >/dev/null 2>&1; then + sha256sum "$1" | awk '{print $1}' + else + lw_die "neither shasum nor sha256sum is available on this host" + fi +} + +# The fingerprint of the kubeconfig the user already had. The recipe never +# writes to it, and the recorded value is compared again before cleanup so a +# change made elsewhere is reported rather than silently accepted. +lw_default_kubeconfig_fingerprint() { + local path + path="$(lw_default_kubeconfig_path)" + if [[ -f "${path}" ]]; then + lw_sha256 "${path}" + else + printf 'absent\n' + fi +} + +lw_default_kubeconfig_unchanged() { + local recorded current + recorded="$(lw_ownership_read DEFAULT_KUBECONFIG_SHA256)" || return 1 + current="$(lw_default_kubeconfig_fingerprint)" + [[ "${recorded}" == "${current}" ]] +} + +# Ownership record + +lw_ownership_write() { + local node_container_id="$1" node_image_id="$2" file tmp + # A record with an empty identifier refuses every later cleanup, because no + # live container can match it, so an unreadable identifier stops the write. + [[ -n "${node_container_id}" ]] \ + || lw_die "the node container identifier could not be read, the ownership record is not written" + [[ -n "${node_image_id}" ]] \ + || lw_die "the node image identifier could not be read, the ownership record is not written" + file="$(lw_state_file)" + mkdir -p "$(lw_state_dir)" + tmp="${file}.tmp.$$" + { + printf 'CLUSTER_NAME=%s\n' "${LW_CLUSTER_NAME}" + printf 'CONTEXT=%s\n' "$(lw_context)" + printf 'NODE_CONTAINER_NAME=%s\n' "$(lw_node_container_name)" + printf 'NODE_CONTAINER_ID=%s\n' "${node_container_id}" + printf 'KUBECONFIG=%s\n' "$(lw_kubeconfig_path)" + printf 'KUBECTL=%s\n' "$(lw_kubectl_path)" + printf 'NODE_IMAGE=%s\n' "${LW_NODE_IMAGE}" + printf 'NODE_IMAGE_ID=%s\n' "${node_image_id}" + printf 'KUBERNETES_VERSION=%s\n' "${LW_KUBERNETES_VERSION}" + printf 'DEFAULT_KUBECONFIG_SHA256=%s\n' "$(lw_default_kubeconfig_fingerprint)" + printf 'CREATED_AT=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + } >"${tmp}" + mv "${tmp}" "${file}" + lw_log "ownership record written to ${file}" +} + +lw_ownership_read() { + local key="$1" file line k v + file="$(lw_state_file)" + [[ -f "${file}" ]] || return 1 + while IFS= read -r line || [[ -n "${line}" ]]; do + [[ -z "${line}" || "${line}" == \#* ]] && continue + [[ "${line}" == *=* ]] || lw_die "ownership record ${file} is corrupt: a line carries no key" + k="${line%%=*}" + v="${line#*=}" + case " ${LW_OWNERSHIP_KEYS} " in + *" ${k} "*) ;; + *) lw_die "ownership record ${file} is corrupt: unknown key ${k}" ;; + esac + if [[ "${k}" == "${key}" ]]; then + printf '%s\n' "${v}" + return 0 + fi + done <"${file}" + return 1 +} + +# Report whether the record can be read. A record that is present but damaged +# is a hard failure for a caller that acts on it, and a plain refusal for a +# caller that only reports. +lw_ownership_is_readable() { + local file line k + file="$(lw_state_file)" + [[ -f "${file}" ]] || return 1 + while IFS= read -r line || [[ -n "${line}" ]]; do + [[ -z "${line}" || "${line}" == \#* ]] && continue + [[ "${line}" == *=* ]] || return 1 + k="${line%%=*}" + case " ${LW_OWNERSHIP_KEYS} " in + *" ${k} "*) ;; + *) return 1 ;; + esac + done <"${file}" + return 0 +} + +# Returns zero when the record is present and complete. A record that is +# present but damaged is a hard failure: a caller must not treat it as an +# absent record and delete a cluster it cannot prove it owns. +lw_ownership_validate() { + local file key + file="$(lw_state_file)" + [[ -f "${file}" ]] || return 1 + lw_ownership_is_readable || lw_die "the ownership record ${file} is damaged" + for key in ${LW_OWNERSHIP_KEYS}; do + lw_ownership_read "${key}" >/dev/null || lw_die "the ownership record ${file} is missing ${key}" + done + return 0 +} + +lw_ownership_clear() { + rm -f "$(lw_state_file)" +} + +# Pure predicates + +# A cluster name is task-owned when it is well formed and matches the name +# this run recorded. A name that another run created is refused. +lw_cluster_name_valid() { + local name="$1" + [[ "${name}" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] +} + +# A cluster name is free when no existing cluster carries it. +lw_cluster_name_is_free() { + local existing="$1" name="$2" + ! grep -Fx -- "${name}" <<<"${existing}" >/dev/null +} + +# The recorded node container and the live one are the same cluster only when +# the immutable container identifier matches. +lw_ownership_matches() { + local recorded="$1" live="$2" + [[ -n "${recorded}" && -n "${live}" && "${recorded}" == "${live}" ]] +} + +# The recipe runs against its own kubeconfig. A kubeconfig value that points +# anywhere else would send a command to a cluster this run does not own. +lw_kubeconfig_isolated() { + local kubeconfig="$1" state_dir entry + state_dir="$(lw_state_dir)" + if [[ -z "${kubeconfig}" ]]; then + return 0 + fi + local old_ifs="${IFS}" + IFS=':' + for entry in ${kubeconfig}; do + IFS="${old_ifs}" + [[ "${entry}" == "${state_dir}"/* ]] || return 1 + IFS=':' + done + IFS="${old_ifs}" + return 0 +} + +# The recipe exports its own kubeconfig, so every child process stays away +# from the default one. +lw_exported_kubeconfig_is_isolated() { + lw_kubeconfig_isolated "${KUBECONFIG:-}" +} + +# Normalize a rendered container command and args into name=value pairs. +# The rendering is a bracketed list of quoted strings, and a value may itself +# contain a comma, so the items are separated on the quote-comma-quote +# sequence that joins them rather than on the comma alone. +lw_arg_tokens() { + printf '%s\n' "$1" | + sed 's/","/\n/g' | + tr -d '[]"' | + awk '{ + for (i = 1; i <= NF; i++) { + token = $i + gsub(/^--+/, "", token) + if (index(token, "=") > 0) print token + } + }' +} + +lw_has_arg() { + local text="$1" name="$2" value="$3" + lw_arg_tokens "${text}" | grep -Fx -- "${name}=${value}" >/dev/null +} + +# Every extraArgs entry of a kubeadm configuration must be a name and a value. +# The older map shape is a mapping of flag names to values, and a kubeadm +# version that expects the list shape rejects it. +lw_extra_args_use_name_value_lists() { + local file="$1" + awk ' + function indent_of(text, matched) { + match(text, /^[ \t]*/) + return RLENGTH + } + { + line = $0 + if (line ~ /^[ \t]*$/ || line ~ /^[ \t]*#/) next + current_indent = indent_of(line) + if (in_block) { + if (current_indent <= block_indent) { + if (has_name && !has_value) { malformed = 1 } + in_block = 0 + } else { + if (line ~ /^[ \t]*-[[:space:]]*name:/) { + if (has_name && !has_value) { malformed = 1 } + has_name = 1 + has_value = 0 + next + } + if (line ~ /^[ \t]*value:/) { + if (!has_name) { malformed = 1 } + has_value = 1 + next + } + malformed = 1 + next + } + } + if (line ~ /^[ \t]*extraArgs:/) { + in_block = 1 + block_indent = current_indent + blocks++ + has_name = 0 + has_value = 0 + next + } + } + END { + if (in_block && has_name && !has_value) { malformed = 1 } + exit (malformed || blocks == 0) ? 1 : 0 + } + ' "${file}" +} + +# The mirroring controller only copies Endpoints whose Service has no +# selector, and it skips Endpoints that carry the skip-mirror label or the +# leader-election annotation. A fixture that misses any of these conditions +# would produce no slice even with the controller running. +lw_mirror_fixture_is_valid() { + local selector="$1" skip_mirror="$2" leader_annotation="$3" addresses="$4" port_name="$5" + [[ -z "${selector}" ]] || return 1 + [[ -z "${skip_mirror}" ]] || return 1 + [[ -z "${leader_annotation}" ]] || return 1 + [[ -n "${addresses}" ]] || return 1 + [[ -n "${port_name}" ]] || return 1 + return 0 +} + +# Cluster and tool access + +lw_kind() { + kind "$@" +} + +lw_kubectl() { + "$(lw_kubectl_path)" --kubeconfig "$(lw_kubeconfig_path)" --context "$(lw_context)" "$@" +} + +lw_node_container_id() { + docker inspect --format '{{.Id}}' "$1" 2>/dev/null +} + +lw_live_node_container_id() { + lw_node_container_id "$(lw_node_container_name)" +} + +lw_cluster_exists() { + kind get clusters 2>/dev/null | grep -Fx -- "${LW_CLUSTER_NAME}" >/dev/null +} + +# The rendered command and args of a static Pod, which is where the kubeadm +# arguments that were accepted at creation time can be read back. +lw_static_pod_command() { + local pod="$1" + lw_kubectl -n kube-system get pod "${pod}" \ + -o jsonpath='{range .spec.containers[*]}{.command}{" "}{.args}{"\n"}{end}' +} + +lw_static_pod_phase() { + local pod="$1" + lw_kubectl -n kube-system get pod "${pod}" -o jsonpath='{.status.phase}' +} + +# The version the cluster itself reports, read from the server rather than +# assumed from the image that was built. The server version endpoint is used +# because the client and the server versions are reported side by side. +lw_server_version() { + lw_kubectl get --raw /version | awk -F'"' '/"gitVersion"/ {print $4; exit}' +} + +# EndpointSlice observation + +# The names of the EndpointSlices of a service. The status of the API read is +# kept, so a failed read is never read as an empty list, and the read carries +# its own request timeout, so an API that hangs ends as a failed read instead +# of stretching the observation window without bound. The error of the read +# reaches the log, because it names the reason the observation cannot decide. +lw_read_slice_names() { + local service="$1" names status=0 + names="$(lw_kubectl --request-timeout=10s -n "${LW_NAMESPACE}" get endpointslices \ + -l "kubernetes.io/service-name=${service}" -o name)" || status=$? + printf '%s\n' "${names}" + return "${status}" +} + +# Wait for the absence of an EndpointSlice of a service. +# +# The outcome is one of: +# LW_SLICE_ABSENT the window passed and every read succeeded +# LW_SLICE_PRESENT a read succeeded and the slice was there +# LW_SLICE_READ_FAILED a read failed, so the absence is not proven +# +# A read that failed ends the window at once. It is never counted as an +# absence, and the window never reports an absence it did not observe, so a +# window without room for the interval still performs its first read. +lw_wait_for_no_slice() { + local service="$1" timeout="$2" waited=0 names status interval=2 + while (( waited == 0 )) || (( waited < timeout )); do + names="" + status=0 + names="$(lw_read_slice_names "${service}")" || status=$? + if (( status != 0 )); then + lw_log "the EndpointSlice read for ${service} failed with status ${status}, the absence of a mirrored slice is not proven" + return "${LW_SLICE_READ_FAILED}" + fi + if [[ -n "${names}" ]]; then + return "${LW_SLICE_PRESENT}" + fi + lw_sleep "${interval}" + waited=$((waited + interval)) + done + return "${LW_SLICE_ABSENT}" +} + +# Fixture cleanup + +# The fixture namespace belongs to this run. The creation is the proof of that +# ownership: a namespace that already existed was not created here, and the +# recipe never deletes a namespace it did not create. +lw_create_fixture_namespace() { + lw_kubectl create namespace "${LW_NAMESPACE}" +} + +# The fixture namespace belongs to this run. The delete names the task +# kubeconfig and the task context, so it can only reach the cluster this +# recipe owns, and it never fails the caller. +lw_delete_fixture_namespace() { + lw_kubectl delete namespace "${LW_NAMESPACE}" --wait=false >/dev/null 2>&1 || true +} + +# Run a command with the fixture namespace created and its cleanup armed. +# +# The namespace is created first, and the cleanup is armed only after that +# creation succeeded, so a namespace that already existed is never deleted. The +# command runs in a subshell that carries its own exit trap, so the namespace is +# removed when the command fails, the exit status of the failure is the +# command's own, and the traps of the calling shell are left alone. +# +# The trap body keeps the status of the failure: a trap that fails under +# errexit would replace it with the status of the failure of the trap. +lw_with_fixture_cleanup() { + ( + if ! lw_create_fixture_namespace; then + lw_die "the fixture namespace ${LW_NAMESPACE} was not created, the recipe does not adopt a namespace it did not create" + fi + trap 'lw_delete_fixture_namespace || true' EXIT + "$@" + ) +} + +# Provisioning + +lw_download_checksum() { + curl -fsSL "$1" | awk '{print $1}' +} + +# The task-local kubectl is pinned to the same version as the cluster and +# verified against the published checksum. The kubectl the user already has is +# never replaced. +lw_provision_kubectl() { + local os arch base expected target tmp actual + os="$(uname -s | tr '[:upper:]' '[:lower:]')" + arch="$(uname -m)" + case "${arch}" in + arm64 | aarch64) arch="arm64" ;; + x86_64 | amd64) arch="amd64" ;; + *) lw_die "unsupported host architecture ${arch}" ;; + esac + base="https://dl.k8s.io/release/${LW_KUBECTL_VERSION}/bin/${os}/${arch}" + expected="$(lw_download_checksum "${base}/kubectl.sha256")" \ + || lw_die "cannot read the published checksum for kubectl ${LW_KUBECTL_VERSION}" + target="$(lw_kubectl_path)" + mkdir -p "$(dirname "${target}")" + if [[ -x "${target}" ]]; then + actual="$(lw_sha256 "${target}")" + if [[ "${actual}" == "${expected}" ]]; then + lw_log "task-local kubectl ${LW_KUBECTL_VERSION} is present and verified" + return 0 + fi + lw_log "the task-local kubectl does not match the published checksum; downloading it again" + fi + tmp="${target}.download.$$" + curl -fsSL "${base}/kubectl" -o "${tmp}" || lw_die "cannot download kubectl ${LW_KUBECTL_VERSION}" + actual="$(lw_sha256 "${tmp}")" + [[ "${actual}" == "${expected}" ]] \ + || lw_die "the downloaded kubectl has checksum ${actual}, the published checksum is ${expected}" + mv "${tmp}" "${target}" + chmod +x "${target}" + lw_log "task-local kubectl ${LW_KUBECTL_VERSION} installed and verified" +} + +lw_release_tarball_checksum() { + lw_download_checksum \ + "https://dl.k8s.io/release/${LW_KUBERNETES_VERSION}/kubernetes-server-linux-arm64.tar.gz.sha256" +} + +lw_release_tarball_name() { + printf 'kubernetes-server-linux-arm64.tar.gz\n' +} + +# A release tarball that was already downloaded can stand in for the download +# kind performs, which carries a deadline a slow link can exceed. The copy is +# only used when its checksum matches the published one. It lives in a cache +# directory of its own so that removing the state directory keeps it. +lw_release_tarball_path() { + printf '%s\n' "${LOCAL_WORKER_RELEASE_TARBALL:-${LW_REPO_ROOT}/.dist/local-worker-cache/$(lw_release_tarball_name)}" +} + +lw_base_image_index_digest() { + docker buildx imagetools inspect "${LW_BASE_IMAGE}" | + awk '/^Digest:/ {print $2; exit}' +} + +lw_base_image_arm64_digest() { + docker buildx imagetools inspect "${LW_BASE_IMAGE}" | + awk '/^ *Name: *.*@sha256:/ {digest = $NF} /^ *Platform: *linux\/arm64/ {print digest; exit}' +} + +# The node image is a build cache: it is built once and kept, because a +# rebuild costs a full release download. The release build downloads the +# tarball itself; a verified local copy of the same tarball is used instead +# when one is present. +# Whether the cached release tarball still matches the published checksum. +# A cache that no longer matches is reported and ignored: the release build +# below downloads a tarball and verifies it itself, so a stale or partial +# cache never blocks a build. +lw_release_tarball_matches() { + local path="$1" expected actual + expected="$(lw_release_tarball_checksum)" || return 1 + actual="$(lw_sha256 "${path}")" + if [[ "${actual}" != "${expected}" ]]; then + lw_log "the cached release tarball ${path} has checksum ${actual}, the published checksum is ${expected}, the cache is ignored" + return 1 + fi + lw_log "the release tarball ${path} matches the published checksum ${expected}" + return 0 +} + +lw_ensure_node_image() { + local arch tarball + if docker image inspect "${LW_NODE_IMAGE}" >/dev/null 2>&1; then + arch="$(docker image inspect --format '{{.Architecture}}' "${LW_NODE_IMAGE}")" + [[ "${arch}" == "arm64" ]] \ + || lw_die "the node image ${LW_NODE_IMAGE} has architecture ${arch}, not arm64" + lw_log "node image ${LW_NODE_IMAGE} is present" + return 0 + fi + tarball="$(lw_release_tarball_path)" + if [[ -f "${tarball}" ]] && lw_release_tarball_matches "${tarball}"; then + lw_log "building the node image from the verified release tarball" + kind build node-image --type file --arch arm64 \ + --image "${LW_NODE_IMAGE}" \ + --base-image "${LW_BASE_IMAGE}" \ + "${tarball}" + return 0 + fi + lw_log "building the node image for Kubernetes ${LW_KUBERNETES_VERSION} on arm64" + kind build node-image --type release --arch arm64 \ + --image "${LW_NODE_IMAGE}" \ + --base-image "${LW_BASE_IMAGE}" \ + "${LW_KUBERNETES_VERSION}" +} + +lw_node_image_id() { + docker image inspect --format '{{.Id}}' "${LW_NODE_IMAGE}" 2>/dev/null +} diff --git a/testing/infra/local_worker/run.sh b/testing/infra/local_worker/run.sh new file mode 100644 index 000000000..479d8b556 --- /dev/null +++ b/testing/infra/local_worker/run.sh @@ -0,0 +1,670 @@ +#!/usr/bin/env bash +# +# Disposable local cluster recipe for the aggregated API path. +# +# The commands create and remove only the resources this recipe owns: a kind +# cluster with a task-local name, a kubeconfig outside the one the user +# already had, a task-local kubectl, and a node image built locally and kept +# as a build cache. +# +# Every kubectl and kind invocation names the task-local kubeconfig and the +# task-local context, so the default kubeconfig and its current context are +# never read or written. + +set -o errexit +set -o nounset +set -o pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck source-path=SCRIPTDIR +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" + +# The exported value keeps every child process away from the default +# kubeconfig, and the explicit flag on each call keeps that true even when a +# command ignores the environment. +KUBECONFIG="$(lw_kubeconfig_path)" +export KUBECONFIG + +# The controller that copies core Endpoints into EndpointSlices. The exclusion +# token is the alias the controller manager registers for it. +LW_MIRROR_CONTROLLER="endpointslicemirroring" +LW_CONTROLLER_ARG="*,-${LW_MIRROR_CONTROLLER}" + +LW_LISTENER_PID="" +LW_LISTENER_FILES=() + +usage() { + cat <<'USAGE' +Usage: run.sh + +Commands: + preflight Check the host, the tools, the node image and the cluster name. + prepare Build or reuse the node image, create the cluster, prove its shape. + network Prove that a Mac address is reachable from the control plane. + cleanup Remove the cluster and the files this recipe created. +USAGE +} + +lw_stop_listener() { + if [[ -n "${LW_LISTENER_PID}" ]] && kill -0 "${LW_LISTENER_PID}" 2>/dev/null; then + kill "${LW_LISTENER_PID}" 2>/dev/null || true + wait "${LW_LISTENER_PID}" 2>/dev/null || true + fi + LW_LISTENER_PID="" + local file + for file in "${LW_LISTENER_FILES[@]+"${LW_LISTENER_FILES[@]}"}"; do + rm -f "${file}" + done + LW_LISTENER_FILES=() +} + +# Preflight + +lw_check_host() { + local os arch + os="$(uname -s)" + arch="$(uname -m)" + [[ "${os}" == "Darwin" ]] || lw_die "this recipe runs on macOS, this host reports ${os}" + [[ "${arch}" == "arm64" ]] || lw_die "this recipe runs on native arm64, this host reports ${arch}" + lw_log "host is ${os}/${arch}" +} + +lw_check_docker() { + local server_arch + server_arch="$(docker version --format '{{.Server.Arch}}' 2>/dev/null)" \ + || lw_die "the Docker daemon is not reachable" + [[ "${server_arch}" == "arm64" ]] \ + || lw_die "the Docker daemon serves ${server_arch}, the node image is built for arm64" + lw_log "the Docker daemon serves linux/${server_arch}" +} + +lw_check_kind() { + local help + help="$(kind build node-image --help 2>&1)" + grep -q -- '--base-image' <<<"${help}" \ + || lw_die "this kind build has no --base-image flag, so the pinned base image cannot be named" + grep -q -- '--type' <<<"${help}" \ + || lw_die "this kind build has no --type flag, so a release build cannot be requested" + lw_log "kind $(kind version | awk '{print $2}') supports the release build form" +} + +# The cluster name must be free. A name another run already used is refused +# rather than adopted. +lw_check_cluster_name() { + lw_cluster_name_valid "${LW_CLUSTER_NAME}" \ + || lw_die "the cluster name ${LW_CLUSTER_NAME} is not a valid kind cluster name" + local existing + existing="$(kind get clusters 2>/dev/null || true)" + if lw_cluster_name_is_free "${existing}" "${LW_CLUSTER_NAME}"; then + lw_log "the cluster name ${LW_CLUSTER_NAME} is free" + return 0 + fi + if lw_ownership_is_readable \ + && [[ "$(lw_ownership_read CLUSTER_NAME || true)" == "${LW_CLUSTER_NAME}" ]]; then + lw_die "the cluster ${LW_CLUSTER_NAME} already exists and belongs to this recipe; run cleanup first" + fi + lw_die "a cluster named ${LW_CLUSTER_NAME} already exists and this recipe does not own it" +} + +lw_check_node_image() { + if docker image inspect "${LW_NODE_IMAGE}" >/dev/null 2>&1; then + lw_log "the node image ${LW_NODE_IMAGE} is present and is kept as a build cache" + else + lw_log "the node image ${LW_NODE_IMAGE} will be built from the ${LW_BASE_IMAGE} base image" + fi +} + +lw_check_kubeconfig_isolation() { + lw_exported_kubeconfig_is_isolated \ + || lw_die "the exported KUBECONFIG points outside the recipe state directory" +} + +cmd_preflight() { + lw_require_cmd kind docker curl python3 + lw_check_host + lw_check_docker + lw_check_kind + lw_check_cluster_name + lw_check_kubeconfig_isolation + lw_provision_kubectl + lw_check_node_image + lw_log "the release tarball for ${LW_KUBERNETES_VERSION} has checksum $(lw_release_tarball_checksum)" + lw_log "the base image ${LW_BASE_IMAGE} resolves to $(lw_base_image_index_digest)" + lw_log "the base image arm64 manifest is $(lw_base_image_arm64_digest)" + lw_log "the recipe uses the kubeconfig $(lw_kubeconfig_path) and the context $(lw_context)" + lw_log "the default kubeconfig $(lw_default_kubeconfig_path) has the fingerprint $(lw_default_kubeconfig_fingerprint)" + lw_log "preflight finished" +} + +# Prepare + +lw_create_cluster() { + lw_log "creating the cluster ${LW_CLUSTER_NAME}" + if ! kind create cluster \ + --name "${LW_CLUSTER_NAME}" \ + --image "${LW_NODE_IMAGE}" \ + --kubeconfig "$(lw_kubeconfig_path)" \ + --config "${SCRIPT_DIR}/kind-config.yaml" \ + --wait 300s; then + # kind handles its own default failure cleanup unless it is asked to + # retain, and this wrapper issues no deletion of its own: a cluster that + # still carries this name has unproven ownership, whether kind's cleanup + # failed or another run took the name during the window, and a delete by + # name cannot tell those apart. + lw_log "the cluster creation failed; kind handles its own default failure cleanup and this run deletes nothing" + lw_log "a cluster named ${LW_CLUSTER_NAME} that still exists has unproven ownership; inspect it first, and run kind delete cluster --name ${LW_CLUSTER_NAME} only once you have verified it is yours to remove" + lw_die "the cluster ${LW_CLUSTER_NAME} could not be created" + fi +} + +lw_assert_server_version() { + local version + version="$(lw_server_version)" + [[ "${version}" == "${LW_KUBERNETES_VERSION}" ]] \ + || lw_die "the cluster reports ${version}, the recipe builds ${LW_KUBERNETES_VERSION}" + lw_log "the cluster reports ${version}" +} + +lw_assert_node_architecture() { + local node arch + node="$(lw_kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" + arch="$(lw_kubectl get node "${node}" -o jsonpath='{.status.nodeInfo.architecture}')" + [[ "${arch}" == "arm64" ]] || lw_die "the node ${node} reports architecture ${arch}, not arm64" + lw_log "the node ${node} runs ${arch}" +} + +# The arguments the kubeadm configuration asked for are read back from the +# rendered static Pod. The Pod must also be running: an argument the +# component rejects makes it exit, and a component that is not running +# mirrors nothing, which would make the behavioural check below pass for the +# wrong reason. +lw_assert_aggregator_routing() { + local node pod command phase + node="$(lw_kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" + pod="kube-apiserver-${node}" + phase="$(lw_static_pod_phase "${pod}")" + [[ "${phase}" == "Running" ]] || lw_die "the ${pod} static Pod is ${phase}, not Running" + command="$(lw_static_pod_command "${pod}")" + lw_has_arg "${command}" "enable-aggregator-routing" "true" \ + || lw_die "the ${pod} static Pod does not carry enable-aggregator-routing=true" + lw_log "the ${pod} static Pod runs with enable-aggregator-routing=true" +} + +lw_assert_mirroring_controller_disabled() { + local node pod command phase + node="$(lw_kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" + pod="kube-controller-manager-${node}" + phase="$(lw_static_pod_phase "${pod}")" + [[ "${phase}" == "Running" ]] || lw_die "the ${pod} static Pod is ${phase}, not Running" + command="$(lw_static_pod_command "${pod}")" + lw_has_arg "${command}" "controllers" "${LW_CONTROLLER_ARG}" \ + || lw_die "the ${pod} static Pod does not carry controllers=${LW_CONTROLLER_ARG}" + lw_log "the ${pod} static Pod runs with controllers=${LW_CONTROLLER_ARG}" +} + +# The fixture is a Service without a selector and a core Endpoints object +# without the skip-mirror label. The mirroring controller copies such an +# Endpoints into an EndpointSlice, so the fixture has to satisfy every +# condition the controller checks. A fixture that misses one would produce no +# slice even with the controller running. +# +# The namespace is not created here. The cleanup scope creates it before it +# arms the cleanup, so the namespace is deleted only when this recipe created +# it. +lw_create_mirror_fixture() { + local node address + node="$(lw_kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" + address="$(lw_kubectl get node "${node}" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')" + lw_kubectl apply -f - </dev/null | + awk '$1 ~ /pause$/ {print $1 ":" $2; exit}')" + if [[ -z "${image}" ]]; then + image="registry.k8s.io/pause:3.10" + fi + printf '%s\n' "${image}" +} + +lw_fixture_is_valid() { + local selector skip_mirror leader addresses port_name + selector="$(lw_kubectl -n "${LW_NAMESPACE}" get service mirror-check -o jsonpath='{.spec.selector}')" + skip_mirror="$(lw_kubectl -n "${LW_NAMESPACE}" get endpoints mirror-check \ + -o jsonpath='{.metadata.labels.endpointslice\.kubernetes\.io/skip-mirror}')" + leader="$(lw_kubectl -n "${LW_NAMESPACE}" get endpoints mirror-check \ + -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}')" + addresses="$(lw_kubectl -n "${LW_NAMESPACE}" get endpoints mirror-check \ + -o jsonpath='{.subsets[*].addresses[*].ip}')" + port_name="$(lw_kubectl -n "${LW_NAMESPACE}" get endpoints mirror-check \ + -o jsonpath='{.subsets[*].ports[*].name}')" + lw_mirror_fixture_is_valid "${selector}" "${skip_mirror}" "${leader}" "${addresses}" "${port_name}" +} + +# Wait for the EndpointSlice of the control Service. The read discipline is +# the one lw_wait_for_no_slice keeps: a read that fails ends the wait at once +# with LW_SLICE_READ_FAILED, so a control that could not be observed is never +# reported as a control that produced nothing, and a window without room for +# the interval still performs its first read. +lw_wait_for_slice() { + local service="$1" timeout="$2" waited=0 slice status + while (( waited == 0 )) || (( waited < timeout )); do + status=0 + slice="$(lw_read_slice_names "${service}")" || status=$? + if (( status != 0 )); then + lw_log "the EndpointSlice read for ${service} failed with status ${status}, the control cannot be observed" + return "${LW_SLICE_READ_FAILED}" + fi + if [[ -n "${slice}" ]]; then + return 0 + fi + lw_sleep 2 + waited=$((waited + 2)) + done + return 1 +} + +# The mirroring exclusion is proved by observation. The proof runs in the +# fixture cleanup scope, so the fixture namespace is removed when an assertion +# fails, and the exit status of the failure is preserved. +lw_prove_mirroring_disabled() { + lw_with_fixture_cleanup lw_prove_mirroring_disabled_body +} + +lw_prove_mirroring_disabled_body() { + local window=60 observation=0 + lw_create_mirror_fixture + lw_create_slice_control_fixture + lw_kubectl -n "${LW_NAMESPACE}" wait --for=condition=Ready pod/slice-control --timeout=180s >/dev/null + lw_fixture_is_valid \ + || lw_die "the mirror fixture does not satisfy the conditions the mirroring controller checks" + lw_log "the mirror fixture is a selectorless Service with an Endpoints that carries no skip-mirror label" + control=0 + lw_wait_for_slice "slice-control" 120 || control=$? + case "${control}" in + 0) + ;; + "${LW_SLICE_READ_FAILED}") + lw_die "the EndpointSlice read for the control Service failed, so the observation below cannot decide anything" + ;; + *) + lw_die "the control Service produced no EndpointSlice, so the observation below cannot decide anything" + ;; + esac + lw_log "the control Service produced an EndpointSlice, so EndpointSlice creation is working in this cluster" + lw_wait_for_no_slice "mirror-check" "${window}" || observation=$? + case "${observation}" in + "${LW_SLICE_ABSENT}") + lw_log "no mirrored EndpointSlice appeared for the selectorless Service in ${window} seconds" + ;; + "${LW_SLICE_PRESENT}") + lw_die "a mirrored EndpointSlice appeared for the selectorless Service, the mirroring controller is running" + ;; + "${LW_SLICE_READ_FAILED}") + lw_die "the EndpointSlice read failed, so the absence of a mirrored EndpointSlice is not proven" + ;; + *) + lw_die "the EndpointSlice observation returned the unexpected status ${observation}" + ;; + esac + # The success-path delete waits for the fixture namespace to be gone, so + # the caller observes a cluster without it, and the wait itself is bounded, + # so a namespace stuck in Terminating ends the proof instead of hanging it. + lw_kubectl delete namespace "${LW_NAMESPACE}" --wait=true --timeout=120s >/dev/null +} + +cmd_prepare() { + lw_require_cmd kind docker curl python3 + lw_check_host + lw_check_docker + lw_check_kind + lw_check_cluster_name + lw_check_kubeconfig_isolation + lw_provision_kubectl + lw_ensure_node_image + lw_create_cluster + lw_ownership_write "$(lw_live_node_container_id)" "$(lw_node_image_id)" + lw_assert_server_version + lw_assert_node_architecture + lw_assert_aggregator_routing + lw_assert_mirroring_controller_disabled + lw_prove_mirroring_disabled + lw_default_kubeconfig_unchanged \ + || lw_die "the default kubeconfig changed during this run, the recipe must not touch it" + lw_log "prepare finished" +} + +# Network + +# The host address the control plane reaches the Mac through. The name the +# container runtime provides is resolved inside the node, so the address is +# the one the container really uses. +lw_resolve_host_address() { + local node address + node="$(lw_node_container_name)" + address="$(docker exec "${node}" getent ahostsv4 host.docker.internal 2>/dev/null | + awk 'NR == 1 {print $1}')" + if [[ -z "${address}" ]]; then + address="$(docker exec "${node}" awk '/host\.docker\.internal/ {print $1; exit}' /etc/hosts 2>/dev/null)" + fi + [[ -n "${address}" ]] || lw_die "the node cannot resolve host.docker.internal" + printf '%s\n' "${address}" +} + +# The Mac address of the interface that leaves the host, used when the +# container runtime alias is not reachable. +lw_host_interface_address() { + local interface + interface="$(route -n get default 2>/dev/null | awk '/interface:/ {print $2}')" + if [[ -n "${interface}" ]]; then + ipconfig getifaddr "${interface}" 2>/dev/null || true + fi +} + +lw_can_bind() { + python3 - "$1" <<'PY' +import socket +import sys + +sock = socket.socket() +sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) +try: + sock.bind((sys.argv[1], 0)) +except OSError: + sys.exit(1) +finally: + sock.close() +PY +} + +lw_write_listener() { + local path="$1" + cat >"${path}" <<'LISTENER' +import http.server +import socketserver +import sys +import time + +bind_address, port_file, token = sys.argv[1], sys.argv[2], sys.argv[3] + + +class Handler(http.server.BaseHTTPRequestHandler): + def do_GET(self): + body = token.encode() + self.send_response(200) + self.send_header("Content-Type", "text/plain") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, fmt, *args): + pass + + +class Server(socketserver.TCPServer): + allow_reuse_address = True + + +server = Server((bind_address, 0), Handler) +with open(port_file, "w") as handle: + handle.write(str(server.server_address[1])) +# One stray connection must not consume the probe, so requests are served +# until the deadline and the probe carries the token away on its own request. +deadline = time.monotonic() + 120 +while time.monotonic() < deadline: + server.timeout = deadline - time.monotonic() + server.handle_request() +server.server_close() +LISTENER +} + +lw_token() { + if command -v uuidgen >/dev/null 2>&1; then + printf 'gpustack-local-worker-%s\n' "$(uuidgen | tr '[:upper:]' '[:lower:]')" + else + printf 'gpustack-local-worker-%s-%s\n' "$(date +%s)" "$$" + fi +} + +# The probe runs inside the node container. The plain HTTP request carries no +# certificate and no identity, so nothing about the later authenticated worker +# is proved here. +lw_probe_from_node() { + local node="$1" host="$2" port="$3" url + url="http://${host}:${port}/" + if docker exec "${node}" sh -c 'command -v curl' >/dev/null 2>&1; then + docker exec "${node}" curl --fail --silent --show-error --max-time 15 "${url}" + return $? + fi + if docker exec "${node}" sh -c 'command -v wget' >/dev/null 2>&1; then + docker exec "${node}" wget --quiet --timeout=15 --output-document=- "${url}" + return $? + fi + # The raw-socket fallback needs bash with /dev/tcp and coreutils timeout in + # the node image. The pinned image carries both, so this guard is for an + # overridden image: it names what is missing instead of failing mid-probe. + if ! docker exec "${node}" sh -c 'command -v bash >/dev/null 2>&1 && command -v timeout >/dev/null 2>&1'; then + lw_die "the node image carries no HTTP client and no bash with timeout for the raw-socket probe" + fi + # The raw-socket fallback has no deadline of its own, so the timeout + # bounds the whole probe the way the curl and wget paths above are bounded. + docker exec "${node}" timeout 15 bash -c ' + host="$1" + port="$2" + exec 3<>/dev/tcp/"$host"/"$port" + printf "GET / HTTP/1.0\r\nHost: %s\r\n\r\n" "$host" >&3 + cat <&3 + ' _ "${host}" "${port}" +} + +# Start a listener on the Mac, ask the control plane for its token and stop +# the listener again. The listener binds the address itself when the Mac owns +# it, and every Mac interface otherwise. +lw_probe_address() { + local node="$1" address="$2" label="$3" + local bind_address token workdir listener port_file log_file response port waited=0 + workdir="$(lw_state_dir)/network" + mkdir -p "${workdir}" + listener="${workdir}/listener.py" + port_file="${workdir}/port" + log_file="${workdir}/listener.log" + rm -f "${port_file}" "${log_file}" + LW_LISTENER_FILES=("${listener}" "${port_file}" "${log_file}") + + if lw_can_bind "${address}"; then + bind_address="${address}" + else + bind_address="0.0.0.0" + lw_log "the Mac cannot bind ${address}, the listener binds every Mac interface instead" + fi + + token="$(lw_token)" + lw_write_listener "${listener}" + python3 "${listener}" "${bind_address}" "${port_file}" "${token}" >"${log_file}" 2>&1 & + LW_LISTENER_PID=$! + + while [[ ! -s "${port_file}" ]]; do + if ! kill -0 "${LW_LISTENER_PID}" 2>/dev/null; then + lw_stop_listener + return 1 + fi + sleep 1 + waited=$((waited + 1)) + if (( waited > 30 )); then + lw_stop_listener + return 1 + fi + done + port="$(cat "${port_file}")" + + response="$(lw_probe_from_node "${node}" "${address}" "${port}")" || { + lw_stop_listener + return 1 + } + if ! grep -Fq -- "${token}" <<<"${response}"; then + lw_stop_listener + return 1 + fi + + lw_stop_listener + lw_log "the control plane reached ${address}:${port} over plain HTTP and received the expected token" + lw_log "the ${label} probe used http://${address}:${port}/ with the listener bound to ${bind_address}" + return 0 +} + +cmd_network() { + lw_require_cmd docker python3 + lw_ownership_validate || lw_die "the cluster was not created by this recipe, there is nothing to probe" + local node address interface_address + node="$(lw_node_container_name)" + address="$(lw_resolve_host_address)" + lw_log "the node resolves host.docker.internal to ${address}" + + lw_probe_address "${node}" "${address}" "container host alias" \ + || lw_die "the control plane could not reach ${address} from the Mac" + + # The container runtime alias names the gateway the container uses to reach + # the host. The address of a Mac interface is probed as well, with the + # listener bound to that address, so the proof names an address the Mac + # itself owns. + interface_address="$(lw_host_interface_address)" + if [[ -n "${interface_address}" && "${interface_address}" != "${address}" ]]; then + if lw_probe_address "${node}" "${interface_address}" "Mac interface"; then + lw_log "the Mac interface address ${interface_address} is reachable from the control plane" + else + lw_log "the Mac interface address ${interface_address} is not reachable from the control plane" + fi + fi + lw_log "network finished" +} + +# Cleanup + +cmd_cleanup() { + lw_require_cmd kind docker + if ! lw_ownership_validate; then + lw_log "no ownership record was found, so no cluster is proven to belong to this recipe" + lw_log "the cluster ${LW_CLUSTER_NAME} and the files under $(lw_state_dir) were left untouched" + return 0 + fi + local recorded live + if [[ "$(lw_ownership_read CLUSTER_NAME)" != "${LW_CLUSTER_NAME}" ]]; then + lw_die "the ownership record names cluster $(lw_ownership_read CLUSTER_NAME), not ${LW_CLUSTER_NAME}, so the record is kept: the cluster it names may still be running" + fi + recorded="$(lw_ownership_read NODE_CONTAINER_ID)" + live="$(lw_live_node_container_id)" + if [[ -z "${live}" ]]; then + lw_log "the node container is already gone, only the recipe files are removed" + rm -rf "$(lw_state_dir)" + return 0 + fi + if ! lw_ownership_matches "${recorded}" "${live}"; then + lw_die "the live node container ${live} is not the one this recipe created, the cluster is not removed" + fi + lw_log "removing the cluster ${LW_CLUSTER_NAME}" + kind delete cluster --name "${LW_CLUSTER_NAME}" --kubeconfig "$(lw_kubeconfig_path)" + lw_default_kubeconfig_unchanged \ + || lw_die "the default kubeconfig changed during this cleanup, the recipe must not touch it" + rm -rf "$(lw_state_dir)" + lw_log "the node image ${LW_NODE_IMAGE} is kept as a build cache" + lw_log "cleanup finished" +} + +case "${1:-}" in + preflight) + cmd_preflight + ;; + prepare) + cmd_prepare + ;; + network) + cmd_network + ;; + cleanup) + cmd_cleanup + ;; + verify) + lw_die "the verify command is not part of this recipe, it starts the authenticated worker and belongs to a later stage" + ;; + *) + usage + exit 1 + ;; +esac diff --git a/testing/infra/local_worker/test_recipe.sh b/testing/infra/local_worker/test_recipe.sh new file mode 100644 index 000000000..bc607d607 --- /dev/null +++ b/testing/infra/local_worker/test_recipe.sh @@ -0,0 +1,535 @@ +#!/usr/bin/env bash +# +# Safety and validation cases for the local cluster recipe. +# +# The cases run in two groups. The fast group calls the pure predicates of +# lib.sh with declarative inputs and needs no cluster. The live group runs the +# recipe end to end: preflight, prepare, network, cleanup, and a second +# cleanup that has to succeed without touching anything. +# +# The live group is skipped when LOCAL_WORKER_TEST_FAST_ONLY is set to true. + +set -o errexit +set -o nounset +set -o pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck source-path=SCRIPTDIR +# shellcheck source=lib.sh +source "${SCRIPT_DIR}/lib.sh" + +PASS_COUNT=0 +FAIL_COUNT=0 +FAILED_CASES=() +TMP_DIR="" +CLUSTER_WAS_PREPARED="false" + +cleanup_tmp() { + if [[ -n "${TMP_DIR}" && -d "${TMP_DIR}" ]]; then + rm -rf "${TMP_DIR}" + fi +} + +cleanup_cluster() { + if [[ "${CLUSTER_WAS_PREPARED}" == "true" ]]; then + # A run that stops between a tamper with the ownership record and its + # restore leaves the damaged record on disk, and a cleanup that reads it + # refuses to remove the cluster, so the backup goes back first. + if [[ -f "${OWNERSHIP_BACKUP}" && -f "$(lw_state_file)" ]] \ + && ! cmp -s "${OWNERSHIP_BACKUP}" "$(lw_state_file)"; then + cp "${OWNERSHIP_BACKUP}" "$(lw_state_file)" + fi + bash "${SCRIPT_DIR}/run.sh" cleanup >/dev/null 2>&1 || true + fi +} + +trap 'cleanup_cluster; cleanup_tmp' EXIT + +# Each case names one contract, the value it expects, and an expression that +# prints the observed value. The expression holds the data, not the logic. +run_case() { + local name="$1" expected="$2" actual="$3" + if [[ "${actual}" == "${expected}" ]]; then + printf 'PASS %s\n' "${name}" + PASS_COUNT=$((PASS_COUNT + 1)) + else + printf 'FAIL %s: expected %s, got %s\n' "${name}" "${expected}" "${actual}" + FAIL_COUNT=$((FAIL_COUNT + 1)) + FAILED_CASES+=("${name}") + fi +} + +# The predicate calls below report a word, so a case states a word too and a +# difference is readable without decoding an exit status. The standard input +# is closed so a predicate cannot read the case table. +verdict() { + if "$@" >/dev/null 2>&1 "${TMP_DIR}/map-shaped.yaml" <<'FIXTURE' +apiVersion: kubeadm.k8s.io/v1beta4 +kind: ClusterConfiguration +apiServer: + extraArgs: + "enable-aggregator-routing": "true" +FIXTURE + +cat >"${TMP_DIR}/nameless-value.yaml" <<'FIXTURE' +apiVersion: kubeadm.k8s.io/v1beta4 +kind: ClusterConfiguration +apiServer: + extraArgs: + - value: "true" +FIXTURE + +cat >"${TMP_DIR}/name-without-value.yaml" <<'FIXTURE' +apiVersion: kubeadm.k8s.io/v1beta4 +kind: ClusterConfiguration +apiServer: + extraArgs: + - name: "enable-aggregator-routing" +controllerManager: + extraArgs: + - name: controllers + value: "*,-endpointslicemirroring" +FIXTURE + +cat >"${TMP_DIR}/name-without-value-at-end.yaml" <<'FIXTURE' +apiVersion: kubeadm.k8s.io/v1beta4 +kind: ClusterConfiguration +apiServer: + extraArgs: + - name: "enable-aggregator-routing" +FIXTURE + +cat >"${TMP_DIR}/list-shaped.yaml" <<'FIXTURE' +apiVersion: kubeadm.k8s.io/v1beta4 +kind: ClusterConfiguration +apiServer: + extraArgs: + - name: "enable-aggregator-routing" + value: "true" +FIXTURE + +# Fast cases + +while IFS='|' read -r name expected expression; do + [[ -z "${name}" ]] && continue + run_case "${name}" "${expected}" "$(eval "${expression}")" +done <<'CASES' +a well formed cluster name is accepted|yes|verdict lw_cluster_name_valid gpustack-local-worker +an uppercase cluster name is refused|no|verdict lw_cluster_name_valid GPustack-Local +a cluster name with a space is refused|no|verdict lw_cluster_name_valid "gpustack local" +a cluster name another cluster holds is not free|no|verdict lw_cluster_name_is_free $'kind-alpha\nkind-beta' kind-alpha +a cluster name no cluster holds is free|yes|verdict lw_cluster_name_is_free "kind-alpha" gpustack-local-worker +the same container identifier matches|yes|verdict lw_ownership_matches sha256:aaa sha256:aaa +a different container identifier does not match|no|verdict lw_ownership_matches sha256:aaa sha256:bbb +an empty recorded identifier does not match|no|verdict lw_ownership_matches "" sha256:bbb +a kubeconfig outside the state directory is not isolated|no|verdict lw_kubeconfig_isolated /tmp/other/kubeconfig +an unset kubeconfig is isolated|yes|verdict lw_kubeconfig_isolated "" +a kubeconfig inside the state directory is isolated|yes|verdict lw_kubeconfig_isolated "$(lw_state_dir)/kubeconfig" +the recipe configuration uses name and value lists|yes|verdict lw_extra_args_use_name_value_lists "${SCRIPT_DIR}/kind-config.yaml" +a map shaped configuration is refused|no|verdict lw_extra_args_use_name_value_lists "${TMP_DIR}/map-shaped.yaml" +a value without a name is refused|no|verdict lw_extra_args_use_name_value_lists "${TMP_DIR}/nameless-value.yaml" +a name without a value is refused|no|verdict lw_extra_args_use_name_value_lists "${TMP_DIR}/name-without-value.yaml" +a name without a value at the end of the file is refused|no|verdict lw_extra_args_use_name_value_lists "${TMP_DIR}/name-without-value-at-end.yaml" +a name and value list is accepted|yes|verdict lw_extra_args_use_name_value_lists "${TMP_DIR}/list-shaped.yaml" +a selectorless service with an address and a port is valid|yes|verdict lw_mirror_fixture_is_valid "" "" "" "10.0.0.1" https +a service with a selector is not valid|no|verdict lw_mirror_fixture_is_valid "app=x" "" "" "10.0.0.1" https +endpoints carrying the skip-mirror label are not valid|no|verdict lw_mirror_fixture_is_valid "" "true" "" "10.0.0.1" https +endpoints carrying the leader annotation are not valid|no|verdict lw_mirror_fixture_is_valid "" "" "x" "10.0.0.1" https +endpoints without an address are not valid|no|verdict lw_mirror_fixture_is_valid "" "" "" "" https +endpoints without a named port are not valid|no|verdict lw_mirror_fixture_is_valid "" "" "" "10.0.0.1" "" +the aggregator routing flag is found in a rendered command|yes|verdict lw_has_arg "--enable-aggregator-routing=true --v=2" enable-aggregator-routing true +the mirroring exclusion token is found in a rendered command|yes|verdict lw_has_arg "--controllers=*,-endpointslicemirroring --v=2" controllers "*,-endpointslicemirroring" +a flag with another value is not found|no|verdict lw_has_arg "--enable-aggregator-routing=false" enable-aggregator-routing true +a quoted flag value is found|yes|verdict lw_has_arg '--controllers="*,-endpointslicemirroring"' controllers "*,-endpointslicemirroring" +a flag in a rendered command list is found|yes|verdict lw_has_arg '["kube-apiserver","--enable-aggregator-routing=true","--v=2"]' enable-aggregator-routing true +a flag with a comma in its value is found in a rendered command list|yes|verdict lw_has_arg '["kube-controller-manager","--controllers=*,-endpointslicemirroring"]' controllers "*,-endpointslicemirroring" +a flag in a rendered argument list is found|yes|verdict lw_has_arg '["--enable-aggregator-routing=true","--v=2"]' enable-aggregator-routing true +CASES + +# The EndpointSlice observation cases + +# The name a slice carries when it is there. +STUB_SLICE_NAME="endpointslices.discovery.k8s.io/mirror-check-abcde" +# The outcomes the stub of the read serves, in order. +STUB_SEQUENCE="" +# The file the stub records the reads it served in. +STUB_READ_LOG="${TMP_DIR}/stub-reads.log" + +# The stub of the slice read. It serves the outcomes of the sequence in order +# and records every read it served in the log file. The record is the state of +# the stub, because the library reads the slice inside a command substitution +# and a shell variable would not survive it. The stub is installed inside the +# subshell of a case, so the live group keeps the real read. +stub_slice_read() { + lw_read_slice_names() { + local -a outcomes=() + local index outcome + IFS=',' read -r -a outcomes <<<"${STUB_SEQUENCE}" + index="$(wc -l <"${STUB_READ_LOG}" | tr -d ' ')" + if (( index >= ${#outcomes[@]} )); then + printf 'the stub has no read left to serve\n' >&2 + return 1 + fi + outcome="${outcomes[index]}" + printf '%s\n' "${outcome}" >>"${STUB_READ_LOG}" + case "${outcome}" in + empty) + return 0 + ;; + listed) + printf '%s\n' "${STUB_SLICE_NAME}" + return 0 + ;; + failed) + printf 'the API server refused the read\n' >&2 + return 1 + ;; + *) + printf 'the stub does not know the outcome %s\n' "${outcome}" >&2 + return 1 + ;; + esac + } +} + +# The status one slice read returns. The stub of the API call lives in the +# subshell of the case, so the live group keeps the real call. +slice_read_status() { + local api_status="$1" + ( + lw_kubectl() { + if (( api_status != 0 )); then + printf 'the API server refused the read\n' >&2 + return "${api_status}" + fi + printf '%s\n' "${STUB_SLICE_NAME}" + return 0 + } + local status=0 + lw_read_slice_names mirror-check >/dev/null 2>&1 || status=$? + printf '%s\n' "${status}" + ) +} + +# The names one slice read printed. +slice_read_names() { + ( + lw_kubectl() { + printf '%s\n' "${STUB_SLICE_NAME}" + return 0 + } + lw_read_slice_names mirror-check 2>/dev/null + ) +} + +# The outcome of one observation over a sequence of read outcomes. +slice_observation() { + local sequence="$1" timeout="$2" + ( + lw_sleep() { + : + } + STUB_SEQUENCE="${sequence}" + : >"${STUB_READ_LOG}" + stub_slice_read + local status=0 + lw_wait_for_no_slice mirror-check "${timeout}" >/dev/null 2>&1 || status=$? + case "${status}" in + 0) + printf 'absent\n' + ;; + 1) + printf 'present\n' + ;; + 2) + printf 'read-failed\n' + ;; + *) + printf 'unexpected-%s\n' "${status}" + ;; + esac + ) +} + +# The number of reads one observation made. +slice_observation_reads() { + local sequence="$1" timeout="$2" + ( + lw_sleep() { + : + } + STUB_SEQUENCE="${sequence}" + : >"${STUB_READ_LOG}" + stub_slice_read + lw_wait_for_no_slice mirror-check "${timeout}" >/dev/null 2>&1 || true + ) + wc -l <"${STUB_READ_LOG}" | tr -d ' ' +} + +while IFS='|' read -r name expected expression; do + [[ -z "${name}" ]] && continue + run_case "${name}" "${expected}" "$(eval "${expression}")" +done <<'CASES' +a successful read returns the status of the call|0|slice_read_status 0 +a failed read returns the status of the call|1|slice_read_status 1 +a successful read prints the names it read|endpointslices.discovery.k8s.io/mirror-check-abcde|slice_read_names +a successful absence is reported as absent|absent|slice_observation empty,empty 4 +a nonempty slice is reported as present|present|slice_observation listed 4 +a failed read is reported as a failed read|read-failed|slice_observation failed 4 +a failure after an empty observation is reported as a failed read|read-failed|slice_observation empty,failed 4 +the observation reads no more than the bound allows|2|slice_observation_reads empty,empty,empty 4 +a window without room for the interval still performs its first read|1|slice_observation_reads empty,empty 0 +CASES + +# The fixture cleanup cases + +# The file the stub of the namespace delete records the calls in. +STUB_CLEANUP_LOG="${TMP_DIR}/stub-cleanups.log" + +# The file the cleanup scope reports its exit status in. +STUB_CLEANUP_STATUS_FILE="${TMP_DIR}/stub-cleanup-status" + +# A command that fails, for the cleanup scope to run. +stub_failing_command() { + exit 1 +} + +# A command that succeeds, for the cleanup scope to run. +stub_succeeding_command() { + return 0 +} + +# Run the cleanup scope with the creation and the delete stubbed. The case +# states whether the namespace was created, whether the delete fails, and the +# command the scope runs. +scope_run() { + local create_status="$1" cleanup_status="$2" body="$3" + : >"${STUB_CLEANUP_LOG}" + ( + # shellcheck disable=SC2317 # the scope calls it; the checker cannot see the call + lw_create_fixture_namespace() { + printf 'the namespace already exists\n' >&2 + return "${create_status}" + } + # shellcheck disable=SC2317 # the trap keeps it reachable; the checker cannot see the EXIT + lw_delete_fixture_namespace() { + printf 'deleted\n' >>"${STUB_CLEANUP_LOG}" + return "${cleanup_status}" + } + local status=0 + lw_with_fixture_cleanup "${body}" >/dev/null 2>&1 || status=$? + printf '%s\n' "${status}" >"${STUB_CLEANUP_STATUS_FILE}" + ) +} + +# The exit status the cleanup scope reported. +scope_status() { + local create_status="$1" cleanup_status="$2" body="$3" + scope_run "${create_status}" "${cleanup_status}" "${body}" + cat "${STUB_CLEANUP_STATUS_FILE}" +} + +# The number of times the cleanup scope removed the fixture namespace. +scope_cleanups() { + local create_status="$1" cleanup_status="$2" body="$3" + scope_run "${create_status}" "${cleanup_status}" "${body}" + wc -l <"${STUB_CLEANUP_LOG}" | tr -d ' ' +} + +# Whether the wrapper of a failed cluster creation deleted anything. kind is +# stubbed to fail the creation and to record its calls: kind itself removes +# the resources it created on failure, so the wrapper must not delete by +# name, because a foreign cluster that took the name meanwhile would be +# removed by that delete. The wrapper is read out of run.sh, so the case +# tests the source the recipe runs. +create_failure_deletes() { + local fragment="${TMP_DIR}/create-cluster.fn" calls="${TMP_DIR}/kind-calls.log" + sed -n '/^lw_create_cluster() {/,/^}/p' "${SCRIPT_DIR}/run.sh" >"${fragment}" + : >"${calls}" + ( + # shellcheck disable=SC1090 # the fragment holds the wrapper under test + source "${fragment}" + # shellcheck disable=SC2317 # the wrapper calls it; the checker cannot see the call + kind() { + printf '%s\n' "$*" >>"${calls}" + [[ "$1" == "create" ]] && return 1 + return 0 + } + lw_create_cluster >/dev/null 2>&1 || true + ) + if grep -q '^delete cluster' "${calls}"; then + printf 'deleted\n' + else + printf 'none\n' + fi +} + +# The exit trap of the shell that runs the cleanup scope. The stubs are +# defined in this shell, not in a nested subshell, because the case must +# compare the trap table of the very shell that calls the scope: a scope that +# armed its trap in the caller would change this table, and a scope that +# isolates itself in a subshell leaves it alone. The stub definitions stay +# inside the case because the case runs in its own command substitution. +scope_trap() { + local before after + before="$(trap -p EXIT)" + # shellcheck disable=SC2317 # the scope calls it; the checker cannot see the call + lw_create_fixture_namespace() { + return 0 + } + # shellcheck disable=SC2317 # the trap keeps it reachable; the checker cannot see the EXIT + lw_delete_fixture_namespace() { + return 0 + } + lw_with_fixture_cleanup stub_failing_command >/dev/null 2>&1 || true + after="$(trap -p EXIT)" + if [[ "${before}" == "${after}" ]]; then + printf 'unchanged\n' + else + printf 'changed\n' + fi +} + +while IFS='|' read -r name expected expression; do + [[ -z "${name}" ]] && continue + run_case "${name}" "${expected}" "$(eval "${expression}")" +done <<'CASES' +the fixture namespace is removed when the proof fails|1|scope_cleanups 0 0 stub_failing_command +the failure status survives a failing cleanup|1|scope_status 0 3 stub_failing_command +the scope fails when the namespace was not created|1|scope_status 1 0 stub_succeeding_command +a namespace that was not created is not deleted|0|scope_cleanups 1 0 stub_succeeding_command +the cleanup scope leaves the trap of the calling shell alone|unchanged|scope_trap +a failed cluster creation deletes nothing by name|none|create_failure_deletes +CASES + +# Live cases + +# A live case names the contract, the exit status it expects, and the command +# that produces the observed status. The command output is printed only when +# the case fails. +run_live() { + local name="$1" expected="$2" + shift 2 + local status=0 output + output="$("$@" 2>&1)" || status=$? + if [[ "${status}" == "${expected}" ]]; then + printf 'PASS %s\n' "${name}" + PASS_COUNT=$((PASS_COUNT + 1)) + else + printf 'FAIL %s: expected exit %s, got %s\n' "${name}" "${expected}" "${status}" + printf '%s\n' "${output}" | tail -25 + FAIL_COUNT=$((FAIL_COUNT + 1)) + FAILED_CASES+=("${name}") + fi +} + +if [[ "${LOCAL_WORKER_TEST_FAST_ONLY:-false}" == "true" ]]; then + printf '\nThe live group is skipped because LOCAL_WORKER_TEST_FAST_ONLY is true.\n' +else + printf '\n--- live group ---\n' + + run_live "preflight succeeds" 0 bash "${SCRIPT_DIR}/run.sh" preflight + + # The flag goes up before prepare runs: a prepare that fails after the + # cluster was created still has its cleanup taken by the exit trap, because + # the trap is the last chance to remove a cluster this run created. + CLUSTER_WAS_PREPARED="true" + run_live "prepare succeeds" 0 bash "${SCRIPT_DIR}/run.sh" prepare + + # The expressions in this table are read by the shared loop and evaluated + # there, so they expand at evaluation time only. + while IFS='|' read -r name expected expression; do + [[ -z "${name}" ]] && continue + run_case "${name}" "${expected}" "$(eval "${expression}")" + done <<'CASES' +the ownership record exists|yes|[[ -f "$(lw_state_file)" ]] && printf 'yes\n' || printf 'no\n' +the cluster exists|yes|lw_cluster_exists && printf 'yes\n' || printf 'no\n' +the task-local kubeconfig exists|yes|[[ -f "$(lw_kubeconfig_path)" ]] && printf 'yes\n' || printf 'no\n' +the default kubeconfig does not name the task context|no|grep -Fq -- "$(lw_context)" "$(lw_default_kubeconfig_path)" 2>/dev/null && printf 'yes\n' || printf 'no\n' +the default kubeconfig is unchanged|yes|lw_default_kubeconfig_unchanged && printf 'yes\n' || printf 'no\n' +the fixture namespace is gone after prepare|no|lw_kubectl get namespace "${LW_NAMESPACE}" >/dev/null 2>&1 && printf 'yes\n' || printf 'no\n' +CASES + + # The pinned versions are read from the configuration, so a run that + # overrides them compares against what it asked for. + run_case "the task-local kubectl is the pinned version" "${LW_KUBECTL_VERSION}" \ + "$("$(lw_kubectl_path)" version --client -o json | awk -F'"' '/"gitVersion"/ {print $4; exit}')" + run_case "the cluster reports the pinned server version" "${LW_KUBERNETES_VERSION}" "$(lw_server_version)" + + # A namespace that already existed must survive the proof. The namespace is + # one this run created for the check, so the collision is exercised without + # touching a namespace the recipe did not create, and the teardown removes + # only that namespace. + foreign_namespace="gpustack-local-worker-foreign-$$" + cat >"${TMP_DIR}/collision.sh" <<'COLLISION' +#!/usr/bin/env bash +set -o errexit +set -o nounset +set -o pipefail +source "${1}/lib.sh" +lw_with_fixture_cleanup true +COLLISION + lw_kubectl create namespace "${foreign_namespace}" + run_live "the proof refuses a namespace it did not create" 1 \ + env LOCAL_WORKER_NAMESPACE="${foreign_namespace}" bash "${TMP_DIR}/collision.sh" "${SCRIPT_DIR}" + run_case "the namespace that was not created survived" "yes" \ + "$(lw_kubectl get namespace "${foreign_namespace}" >/dev/null 2>&1 && printf 'yes\n' || printf 'no\n')" + lw_kubectl delete namespace "${foreign_namespace}" + + run_live "network succeeds" 0 bash "${SCRIPT_DIR}/run.sh" network + + # A cluster whose live node is not the one this recipe created must not be + # removed, even when the name matches. + cp "$(lw_state_file)" "${OWNERSHIP_BACKUP}" + sed 's|^NODE_CONTAINER_ID=.*|NODE_CONTAINER_ID=sha256:0000000000000000000000000000000000000000000000000000000000000000|' \ + "${OWNERSHIP_BACKUP}" >"$(lw_state_file)" + run_live "cleanup refuses a cluster it cannot prove it created" 1 \ + bash "${SCRIPT_DIR}/run.sh" cleanup + run_case "the refused cleanup left the cluster alone" "yes" \ + "$(lw_cluster_exists && printf 'yes\n' || printf 'no\n')" + + # A damaged ownership record must not be read as an absent one. + printf 'CLUSTER_NAME=%s\nNOT_A_RECORDED_KEY=value\n' "${LW_CLUSTER_NAME}" >"$(lw_state_file)" + run_live "cleanup refuses a damaged ownership record" 1 \ + bash "${SCRIPT_DIR}/run.sh" cleanup + run_case "the refused cleanup left the cluster alone" "yes" \ + "$(lw_cluster_exists && printf 'yes\n' || printf 'no\n')" + + cp "${OWNERSHIP_BACKUP}" "$(lw_state_file)" + + # A cleanup that names a cluster the record does not name must not delete + # the record, because the cluster the record names may still be running. + run_live "cleanup refuses a cluster name it did not record" 1 \ + env LOCAL_WORKER_CLUSTER_NAME="gpustack-local-worker-other" bash "${SCRIPT_DIR}/run.sh" cleanup + run_case "the name-mismatched cleanup left the cluster alone" "yes" \ + "$(lw_cluster_exists && printf 'yes\n' || printf 'no\n')" + + run_live "cleanup succeeds" 0 bash "${SCRIPT_DIR}/run.sh" cleanup + CLUSTER_WAS_PREPARED="false" + run_case "the cluster is gone" "no" \ + "$(lw_cluster_exists && printf 'yes\n' || printf 'no\n')" + run_case "the ownership record is gone" "no" \ + "$([[ -f "$(lw_state_file)" ]] && printf 'yes\n' || printf 'no\n')" + + # A second cleanup has to succeed without removing anything. + run_live "cleanup is repeatable" 0 bash "${SCRIPT_DIR}/run.sh" cleanup + run_case "the repeat left the cluster alone" "no" \ + "$(lw_cluster_exists && printf 'yes\n' || printf 'no\n')" +fi + +printf '\n%s passed, %s failed\n' "${PASS_COUNT}" "${FAIL_COUNT}" +if (( FAIL_COUNT > 0 )); then + printf 'failed cases:\n' + printf ' %s\n' "${FAILED_CASES[@]}" + exit 1 +fi