Repository navigation
feat(cluster): step a server down and back up under memory pressure without losing its etcd vote #100
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Mesh integration | |
| # Brings up three k3s nodes in Docker and joins them through Headscale with the cluster role (tests/mesh/run.sh), once per scenario: headscale_hosted, headscale_in_cluster, and the headscale_existing policy check, plus the role's refusal of a Docker Compose release that recreates the containers it has just built. headscale_in_cluster also runs with github_runner_cluster_docker_context pinned while the current Docker context points at nothing, which covers the cluster role's own docker calls and the recovery playbook's, and context_checks checks the role refuses a pin it cannot honour. regrade turns memory-pressure regrading on and steps a server down to etcd-only under (simulated) pressure and back up, checking that it keeps its etcd seat and vote and that no other server steps meanwhile. Only on changes that can affect how nodes join the mesh. | |
| # | |
| # Runs on GitHub-hosted ubuntu-latest, whose Docker daemon runs the privileged containers the test needs (k3s, and Tailscale inside it). The work directory sits under runner.temp, which the job always has write access to. | |
| on: | |
| pull_request: | |
| paths: | |
| - roles/github_runner_cluster/** | |
| - plugins/filter/** | |
| - docker-compose.yml | |
| - k3s/** | |
| - playbooks/recover_in_cluster_mesh.yml | |
| - tests/mesh/** | |
| - tests/lib/** | |
| - .github/workflows/mesh-integration.yml | |
| workflow_dispatch: | |
| concurrency: | |
| group: mesh-integration-${{ github.ref }} | |
| cancel-in-progress: true | |
| permissions: | |
| contents: read | |
| jobs: | |
| mesh: | |
| name: Mesh (${{ matrix.scenario }}, Compose ${{ matrix.compose }}${{ matrix.pin && ', context pinned' || '' }}) | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 45 | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| # The hosted scenario also runs on Compose 2.39.0, the first release after the range the cluster role refuses (docker/compose#13047), so its rerun check proves that release keeps every node running. compose_faulty runs on 2.38.2, the last release in that range, and checks the role refuses it. | |
| include: | |
| - {scenario: existing, compose: v5.5.1} | |
| - {scenario: hosted, compose: v5.5.1} | |
| - {scenario: in_cluster, compose: v5.5.1} | |
| - {scenario: regrade, compose: v5.5.1} | |
| - {scenario: hosted, compose: v2.39.0} | |
| - {scenario: compose_faulty, compose: v2.38.2} | |
| - {scenario: in_cluster, compose: v5.5.1, pin: true} | |
| - {scenario: context_checks, compose: v5.5.1} | |
| steps: | |
| - uses: actions/checkout@v7 | |
| - name: Install Docker Compose ${{ matrix.compose }} | |
| # Always the matrix's version, not whatever the runner has, so a runner image update never changes what a scenario tests. The user plugin directory takes precedence over the system one. | |
| env: | |
| COMPOSE_VERSION: ${{ matrix.compose }} | |
| run: | | |
| set -euo pipefail | |
| docker version --format 'Docker Engine {{.Server.Version}}' | |
| case "$(uname -m)" in | |
| x86_64) arch=x86_64 ;; | |
| aarch64 | arm64) arch=aarch64 ;; | |
| *) echo "Unsupported architecture: $(uname -m)" >&2; exit 1 ;; | |
| esac | |
| mkdir -p "${HOME}/.docker/cli-plugins" | |
| base="https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-linux-${arch}" | |
| curl -fsSL -o "${RUNNER_TEMP}/docker-compose" "$base" | |
| echo "$(curl -fsSL "${base}.sha256" | cut -d' ' -f1) ${RUNNER_TEMP}/docker-compose" | sha256sum -c - | |
| install -m 0755 "${RUNNER_TEMP}/docker-compose" "${HOME}/.docker/cli-plugins/docker-compose" | |
| docker compose version | |
| - name: Install Ansible | |
| run: | | |
| set -euo pipefail | |
| if ! python3 -m venv "${RUNNER_TEMP}/venv" 2>/dev/null; then | |
| sudo apt-get update -qq | |
| sudo apt-get install -y -qq python3-venv | |
| python3 -m venv "${RUNNER_TEMP}/venv" | |
| fi | |
| "${RUNNER_TEMP}/venv/bin/pip" install --quiet --prefer-binary ansible-core | |
| "${RUNNER_TEMP}/venv/bin/ansible-galaxy" collection install -r ansible/requirements.yml -p "${RUNNER_TEMP}/collections" | |
| echo "${RUNNER_TEMP}/venv/bin" >> "$GITHUB_PATH" | |
| - name: Run the ${{ matrix.scenario }} scenario | |
| env: | |
| ANSIBLE_COLLECTIONS_PATH: ${{ runner.temp }}/collections | |
| GRTEST_WORK: ${{ runner.temp }}/grtest-mesh | |
| GRTEST_PIN_CONTEXT: ${{ matrix.pin && '1' || '0' }} | |
| run: tests/mesh/run.sh "${{ matrix.scenario }}" | |
| required-checks: | |
| name: Mesh integration result | |
| needs: [mesh] | |
| if: always() | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| steps: | |
| - name: Fail if any scenario failed or was cancelled | |
| run: | | |
| if [[ "${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}" == "true" ]]; then | |
| echo "One or more scenarios did not succeed:" | |
| echo '${{ toJSON(needs) }}' | |
| exit 1 | |
| fi |