Skip to content
CI

feat(cluster): step a server down and back up under memory pressure without losing its etcd vote #246

Sign in to view logs

feat(cluster): step a server down and back up under memory pressure without losing its etcd vote

feat(cluster): step a server down and back up under memory pressure without losing its etcd vote #246

Workflow file for this run

.github/workflows/ci.yml at 3490687
name: CI
# Lints every shell script, workflow file, YAML file, and Dockerfile in the repo. There is no application test suite here (this repo is fleet infrastructure config, not application code) - these checks are its actual verification surface, matching the README's Build/test/lint section.
#
# Runs on GitHub-hosted runners, which are free for a public repository, so the checks of the fleet's own configuration don't depend on the fleet being up. The fleet's own behaviour is tested by test-arc-runner.yml and test-autoscaler.yml, which target it directly.
#
# required-checks is the single required status check in this repo's ruleset (never the individual job names), so adding or renaming a lint job never needs a ruleset update - matching ExaDev/eslint-config and ExaDev/wire-mesh's own ci.yml.
on:
push:
branches: [main]
pull_request:
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
jobs:
shellcheck:
name: Shellcheck
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Lint every shell script
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq shellcheck
mapfile -t scripts < <(find . -type f -name '*.sh' -not -path './.git/*')
echo "Found ${#scripts[@]} shell script(s)."
if [ "${#scripts[@]}" -gt 0 ]; then
shellcheck "${scripts[@]}"
fi
actionlint:
name: actionlint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Install actionlint
run: bash <(curl -fsSL https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash)
- name: Lint every workflow file
run: ./actionlint -color
yamllint:
name: YAML lint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Lint every YAML file
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq yamllint
yamllint .
hadolint:
name: hadolint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Install hadolint
# Detect arch rather than assume amd64, so the step also works on an arm64 runner such as this project's own fleet. Downloads into the workspace (no sudo, no system path) - actions-runner pods run the workflow user without write access to /usr/local/bin.
run: |
set -euo pipefail
case "$(uname -m)" in
x86_64) hadolint_arch=x86_64 ;;
aarch64|arm64) hadolint_arch=arm64 ;;
*) echo "Unsupported architecture: $(uname -m)" >&2; exit 1 ;;
esac
curl -fsSL -o ./hadolint "https://github.com/hadolint/hadolint/releases/download/v2.15.1/hadolint-linux-${hadolint_arch}"
chmod +x ./hadolint
- name: Lint every Dockerfile
run: |
set -euo pipefail
mapfile -t dockerfiles < <(find . -type f -iname 'Dockerfile*' -not -path './.git/*')
echo "Found ${#dockerfiles[@]} Dockerfile(s)."
if [ "${#dockerfiles[@]}" -gt 0 ]; then
# error threshold, not the default (info): this repo deliberately floats several base images/packages on `latest` rather than pinning (see e.g. ghcr.io/exadev/github-runner:latest), so hadolint's warning-level pinning advice (DL3007/DL3008/DL3018) stays visible in the log without failing every run over an already-made, documented tradeoff.
./hadolint --failure-threshold error "${dockerfiles[@]}"
fi
ansible-syntax-check:
name: Ansible syntax check
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Syntax-check every playbook
# From ansible/ so its ansible.cfg applies, against an example inventory, since this repository carries no fleet's own. Every top-level playbook is checked, so a new one is covered without editing this list.
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq ansible pkg-config
cd ansible
for playbook in ../playbooks/*.yml; do
echo "Syntax-checking ${playbook}"
ansible-playbook -i ../examples/three-server-tailscale/inventory.yml "${playbook}" --syntax-check
done
- name: Build the collection
# Proves galaxy.yml's build_ignore still ships what the roles need at runtime, such as the k3s node's Compose file and build context under roles/github_runner_cluster/files.
run: |
set -euo pipefail
ansible-galaxy collection build --output-path "${RUNNER_TEMP}/collection"
tar -tzf "${RUNNER_TEMP}"/collection/*.tar.gz | grep -Fx roles/github_runner_cluster/files/docker-compose.yml
tar -tzf "${RUNNER_TEMP}"/collection/*.tar.gz | grep -Fx roles/github_runner_cluster/files/k3s/node-entrypoint.sh
for file in Dockerfile fleet-regrade.sh fleet-regrade-core.sh fleet-regrade-docker.sh; do
tar -tzf "${RUNNER_TEMP}"/collection/*.tar.gz | grep -Fx "roles/github_runner_cluster/files/regrade/${file}"
done
unit-tests:
name: Unit tests (shard ${{ matrix.shard }})
runs-on: ubuntu-latest
# Each module is run on its own by discovery, dealt out by size so that the shards take about the same amount of work: the tests that run Ansible playbooks cost a process each, and the suite no longer fits one job.
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3]
steps:
- uses: actions/checkout@v7
- name: Install Ansible
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq ansible pkg-config
- name: Unit-test the collection
# Ubuntu's ansible package puts ansible-core on the system Python the filter plugins import from. The runner image can carry another Ansible earlier on PATH, so the tests run with /usr/bin first and ANSIBLE_PLAYBOOK pointing at Ubuntu's, which makes every playbook a test runs use the same ansible-core as the filters it imports; both versions are printed for the record. A module is a shard's when its position in the list of modules, largest first, leaves this shard's index modulo the number of shards.
env:
SHARD: ${{ matrix.shard }}
SHARDS: ${{ strategy.job-total }}
run: |
set -euo pipefail
export PATH="/usr/bin:${PATH}" ANSIBLE_PLAYBOOK=/usr/bin/ansible-playbook
ansible-playbook --version | head -n 1
python3 -c 'import ansible.release; print("ansible-core", ansible.release.__version__)'
mapfile -t modules < <(find tests/unit -maxdepth 1 -name 'test_*.py' -printf '%s %p\n' | LC_ALL=C sort -k1,1nr -k2,2 | cut -d' ' -f2- | awk -v shard="$SHARD" -v shards="$SHARDS" '(NR - 1) % shards == shard')
echo "Shard ${SHARD} of ${SHARDS}: ${#modules[@]} module(s)"
for module in "${modules[@]}"; do
python3 -m unittest discover -s tests/unit -p "$(basename "$module")" -v
done
ansible-lint:
name: ansible-lint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Install ansible-lint
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq python3-pip
python3 -m pip install --user --break-system-packages 'ansible-lint>=25'
# pip --user puts the console script in ~/.local/bin, which isn't on the runner's PATH by default.
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Lint the collection
# playbooks/collections/ansible_collections/exadev/github_runner is a symlink back to this checkout's own root, for local FQCN resolution (see ansible/ansible.cfg's comment on roles_path). Excluding it stops ansible-lint's own directory walk from following that symlink back into the tree it is already linting, which otherwise relints the whole collection a second time under a growing path on every recursion.
run: ansible-lint --profile production --exclude playbooks/collections
collection-build:
name: Collection build check
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- name: Install Ansible
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq ansible
- name: Build the collection tarball
run: ansible-galaxy collection build --force
- name: Install the built tarball and verify its content
run: |
set -euo pipefail
tarball=$(ls exadev-github_runner-*.tar.gz)
install_path="$(mktemp -d)"
ansible-galaxy collection install "$tarball" -p "$install_path"
export ANSIBLE_COLLECTIONS_PATH="$install_path"
echo "Roles shipped in the built collection:"
ls "$install_path/ansible_collections/exadev/github_runner/roles"
echo "Filter plugins shipped in the built collection:"
ansible-doc -t filter -l exadev.github_runner | cat
commitlint:
name: commitlint
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: actions/setup-node@v7
with:
node-version: '22'
cache: npm
- run: npm ci
# Runs the repo's own pinned commitlint rather than a Docker-based action: container actions need the runner to pull an image, which the self-hosted runners can't always do.
- name: Lint every commit in the pull request
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: npx --no-install commitlint --from "$BASE_SHA" --to "$HEAD_SHA" --verbose
commitlint-pr-title:
name: Lint PR title
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: '22'
cache: npm
- run: npm ci
# Rebase-merges keep each commit's own message, so the title isn't what lands on main; it's still checked so the PR list reads consistently. Passed through env, never interpolated into the script.
- name: Lint the pull request title
env:
PR_TITLE: ${{ github.event.pull_request.title }}
run: printf '%s\n' "$PR_TITLE" | npx --no-install commitlint --verbose
required-checks:
name: Required Checks
needs: [shellcheck, actionlint, yamllint, hadolint, ansible-syntax-check, unit-tests, ansible-lint, collection-build, commitlint, commitlint-pr-title]
# Always runs, even if a dependency failed or was skipped, so it can report that failure itself rather than being skipped - a required check that's silently absent blocks a PR indefinitely instead of failing it visibly.
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Fail if any required job failed or was cancelled
run: |
if [[ "${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
echo "One or more required jobs did not succeed:"
echo '${{ toJSON(needs) }}'
exit 1
fi