From 9a9de1d59caeba675b3d1721441aeecf42fb8fd8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 29 Sep 2026 09:39:53 +0000 Subject: [PATCH] docs(docs): Add a quickstart and the example files it copies docs/QUICKSTART.md walks through setting up a strength match in an engine's own repository. It covers the build script, the workflow, a first run, a sequential test carried across runs and testing a pull request. It ends with the errors a first run is likely to hit and a table of the common inputs. The files it copies are in examples/ until there is an example repository: a build script written for cargo and two caller workflows. tests/test_examples.py holds them to the release this tree is and to the inputs the pinned strength.yml takes. It checks that the guide's copy of the example workflow matches the file. It also runs the build script against a stub cargo, with the relative path batch.yml passes, to check that it builds the commit it is given from an export stamped at extraction time. docs/BUILDING-A-REF.md said git archive stamps files at extraction time by default. It records the commit's time, and tar -m is what stamps them at extraction. The call in .github/workflows/README.md named v0.3.0 and now names the current release. AGENTS.md says the examples move with the self pins on a release branch. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018ccJFsmZ4jAKYmFNgNBh9u --- .github/workflows/README.md | 2 +- AGENTS.md | 3 + README.md | 3 + docs/BUILDING-A-REF.md | 8 +- docs/QUICKSTART.md | 252 ++++++++++++++++++++++++++++++++++ examples/README.md | 14 ++ examples/build_at.sh | 33 +++++ examples/sprt.yml | 35 +++++ examples/strength.yml | 30 +++++ tests/test_examples.py | 262 ++++++++++++++++++++++++++++++++++++ 10 files changed, 638 insertions(+), 4 deletions(-) create mode 100644 docs/QUICKSTART.md create mode 100644 examples/README.md create mode 100755 examples/build_at.sh create mode 100644 examples/sprt.yml create mode 100644 examples/strength.yml create mode 100644 tests/test_examples.py diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 4607155..2675cfe 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -19,7 +19,7 @@ permissions: contents: read jobs: match: - uses: aywrite/mache/.github/workflows/strength.yml@v0.3.0 + uses: aywrite/mache/.github/workflows/strength.yml@v0.6.0 with: build: scripts/build_at.sh candidate: ${{ inputs.candidate }} diff --git a/AGENTS.md b/AGENTS.md index 649da26..93bae24 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,6 +39,9 @@ on it: the reusable workflows pin the actions beside them, those pins have to name the version being released, and `GITHUB_TOKEN` may not write under `.github/workflows/`, so the workflow cannot move them itself. `.github/workflows/README.md` says what the pins are. +The calls shown in `examples/`, `docs/QUICKSTART.md` and +`.github/workflows/README.md` name the same version and move in the same +commit. `tests/test_examples.py` holds them to it. Merging that pull request tags `vX.Y.Z` on the commit that carries the version and starts the release, which refuses a tag that does not match `__version__`, builds the sdist and the wheel, publishes diff --git a/README.md b/README.md index 9726a9e..9413ff6 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,9 @@ one sequential test (SPRT). A repository calls it from a workflow of its own, on its pull requests or by hand, and the result goes in the run's summary. fastchess plays the games. +To set one up in your engine's repository, start with +[the quickstart](docs/QUICKSTART.md). + It suits an engine whose changes are still large enough to show within a few thousand games. A small change can take many runs to settle, and [What hosted runners can measure](https://github.com/aywrite/mache#what-hosted-runners-can-measure) diff --git a/docs/BUILDING-A-REF.md b/docs/BUILDING-A-REF.md index 1d17620..ef430f6 100644 --- a/docs/BUILDING-A-REF.md +++ b/docs/BUILDING-A-REF.md @@ -32,14 +32,16 @@ for an hour, every game is between two copies of one program, and the report says zero elo with no interval wide enough to look wrong. Nothing fails. There is no log line. The only sign is a result that is suspiciously close to nothing. -`git archive` stamps at extraction time by default, and `tar -xm` keeps that -behaviour on the way out, so the plain form is the right one: +`git archive` writes the commit's time into the archive, and a plain `tar -x` +puts it back on every file. `-m` is what stops that, so the files are stamped +with the time they were extracted: ```bash git archive "$sha" | tar -xm -C "$export" ``` -What to avoid is anything that restores the commit's timestamps onto the export. +Without the `-m` the export is stamped with the commit's time, which is the +case above. ## The rest of it diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md new file mode 100644 index 0000000..2a69aef --- /dev/null +++ b/docs/QUICKSTART.md @@ -0,0 +1,252 @@ +# Quickstart + +Setting up a strength match in your engine's own repository. + +When you're done you'll have a workflow you run from the Actions tab. It builds +two versions of your engine, plays them against each other on GitHub's hosted +runners and puts the result in the run summary. + +## Requirements + +- your engine speaks UCI and builds on the hosted Ubuntu runner (x86-64 Linux) +- it's in a GitHub repository with Actions turned on, and you can push to it +- if the repository belongs to an organisation, its Actions settings have to + allow actions and reusable workflows from other repositories + +So far mache has only been run from public repositories. + +## 1. Build script + +mache doesn't build your engine. It calls a script in your repository to do +it, once for each side of the match: + +``` +scripts/build_at.sh +``` + +The script builds the engine as it was at `` and leaves a runnable +file at ``. It's run from the root of your repository's checkout. + +Copy [`examples/build_at.sh`](../examples/build_at.sh) to +`scripts/build_at.sh`. It's written for a Rust engine built with cargo, so +you'll probably need to change the two lines at the bottom (the build command, +and where the binary ends up). For make they might look like: + +```bash +make -C "$src" +cp "${src}/engine" "$binary" +``` + +and for cmake: + +```bash +cmake -S "$src" -B "${src}/build" -DCMAKE_BUILD_TYPE=Release +cmake --build "${src}/build" -j +cp "${src}/build/engine" "$binary" +``` + +Leave the `git archive` line above them alone, including the `-m` on `tar`. +[BUILDING-A-REF.md](BUILDING-A-REF.md) explains why. `git archive` also leaves +out submodules, and Git LFS files come out as pointers, so if your engine needs +either (e.g. a network file kept in LFS) fetch them in the script. + +The runner image already has cargo, make, cmake, gcc and clang. The versions +move when GitHub updates the image, so if you need a particular one, install or +pin it in the script. + +Make the script executable, on disk and in git: + +``` +chmod +x scripts/build_at.sh +git add --chmod=+x scripts/build_at.sh +``` + +Then try it. This builds your last commit, not anything uncommitted: + +``` +scripts/build_at.sh HEAD /tmp/engine +printf 'uci\nquit\n' | /tmp/engine | grep uciok +``` + +## 2. Workflow + +Copy [`examples/strength.yml`](../examples/strength.yml) to +`.github/workflows/strength.yml`: + +```yaml +name: Strength + +on: + workflow_dispatch: + inputs: + candidate: + description: Branch, tag, commit or pull request number to test + type: string + baseline: + description: What to measure it against. The last release if empty + type: string + games: + type: number + default: 500 + +# strength.yml asks for no more than this itself. It keeps anything you add +# to this file read-only too. +permissions: + contents: read + +jobs: + match: + uses: aywrite/mache/.github/workflows/strength.yml@v0.6.0 + with: + build: scripts/build_at.sh + candidate: ${{ inputs.candidate }} + baseline: ${{ inputs.baseline }} + games: ${{ inputs.games }} +``` + +`@v0.6.0` is the version of mache you're running. Pin a release rather than +`main`, so your results don't change under you. You can pin the commit the +release is tagged at instead (`@ # v0.6.0`), but the actions it calls are +pinned by tag either way. + +Commit both files and push them to your default branch. GitHub only shows the +Run workflow button for a workflow that's on the default branch. + +## 3. First run + +1. open the Actions tab and pick Strength +2. press Run workflow, fill in the form and run it + +`candidate` is what to test: a branch, tag, commit or pull request number. +Leave it empty to test the branch you ran the workflow from. + +`baseline` is what to test it against. Empty means your latest release, which +is the highest `v[0-9]*` tag with no `-` in it (so `v1.2.0-rc1` is passed +over). If you haven't tagged one yet the match is skipped with this notice: + +``` +No previous release to compare against, skipping +``` + +Until you have a release, put a branch or a commit in `baseline`. + +`games` defaults to 500. Try 100 for the first run. It's enough to show the +build works and the engines play. + +The run starts with a short job that works out the two commits, then five shard +jobs that each build both engines and play a fifth of the games, then one job +that pools them. The report is on the run's page, under the job graph. After a +line naming the two commits it looks something like this (from a 150 game +run): + +``` ++56 ±37 Elo (150 games) +``` + +and then the pair counts, a row per shard, how the games ended and each +engine's clock and node counts. [The README](../README.md#match-estimate) goes +through each part. Each shard also uploads its games, fastchess's own output +and a manifest as an artifact called `strength---shard-`. + +Don't expect much from 500 games. The margin is about ±30 normalized elo, which +is enough to see a big change and not a lot else. (The headline figure is +logistic elo, so the two aren't directly comparable.) +[What hosted runners can measure](../README.md#what-hosted-runners-can-measure) +has numbers for bigger matches. Time your first run too, since how long a match +takes depends on your engine, the time control and the runner. + +## SPRT + +For smaller changes you'll want a sequential test (SPRT) instead. It keeps +playing until it can say whether the change is at least a given amount better. +[`examples/sprt.yml`](../examples/sprt.yml) turns it on: + +```yaml + with: + build: scripts/build_at.sh + candidate: ${{ inputs.candidate }} + baseline: ${{ inputs.baseline }} + sprt: true + elo0: 0 + elo1: 10 + games: 500 + batches: 4 + prior_pairs: ${{ inputs.prior_pairs }} +``` + +- `elo0` and `elo1` are the bounds, in logistic elo +- `games` is the size of one batch +- `batches: 4` lets one run play up to four batches, and it stops as soon as the + test passes or fails + +If four batches aren't enough the verdict is inconclusive, and it ends with +something like: + +``` +Launch another batch with prior_pairs set to 4,18,72,38,18. +``` + +Run the workflow again with those numbers in `prior_pairs` to carry on the same +test. mache doesn't store anything between runs, so if you lose the numbers +you have to start over. + +Keep the candidate, baseline and bounds the same from one run to the next, and +give both sides as commits rather than branch names. A branch can move between +runs, and so can an empty baseline when you tag a new release. + +With more than one batch the artifacts are called +`strength---batch--shard-`. + +## Testing a pull request + +Put the pull request number in `candidate`. An empty baseline still means your +latest release, not the branch the pull request is against, so set `baseline` +to `main` (or whatever it targets) if that's the comparison you want. + +This builds and runs the pull request's code. If it comes from a fork, read it +first, more so if you've set `cache_paths`, since a cache saved by that run can +be restored by later ones. + +A `pull_request` workflow can't pass a candidate or a baseline. mache won't +resolve a ref you name unless the workflow was started by `workflow_dispatch`, +`push`, `schedule` or `release`, where whoever picked the ref can already push +to the repository. +[actions/resolve-ref](../actions/resolve-ref/README.md#the-refusal-and-what-it-does-not-cover) +has the details. + +## When it goes wrong + +- no Run workflow button: the workflow isn't on the default branch yet +- `No previous release to compare against, skipping`: see `baseline` above +- `cannot resolve candidate ...` (or `baseline`): the ref doesn't exist, or the + workflow was started by a trigger other than the four above +- the build step fails with your script's own output: run it locally as in step + 1 +- `scripts/build_at.sh left no runnable tools/new`: the script finished without + leaving an executable file where it was told to + +## Options + +The ones you're most likely to want. The rest are described in +[strength.yml](../.github/workflows/strength.yml). + +| input | default | what it does | +| --- | --- | --- | +| `games` | 500 | games per run (per batch in an SPRT), split between the shards | +| `shards` | 5 | how many jobs to split them across | +| `time_control` | 10+0.1 | passed to fastchess as `tc` | +| `book` | `8moves_v3` | the opening book; `UHO_4060_v2` is the other one mache knows | +| `book_table` | empty | your own book table, for any other book (the contract is in [actions/setup](../actions/setup/README.md#the-book-table-contract)) | +| `cache_paths`, `cache_key` | empty | paths kept between runs with `actions/cache`; with the example script that's `~/.cargo/registry` and `~/.cargo/git`, since it builds in a fresh directory each time | +| `hash_mb` | 256 | the hash size each engine is asked for | +| `concurrency` | 2 | games a shard plays at once | + +## Next + +- [calibrate.yml](../.github/workflows/README.md#the-other-two-contracts) places + your engine on a rating list by playing a gauntlet against rated opponents. It + needs a table of opponents as well as the build script. +- if you want your own cache action, build steps or manifest, call the composite + actions from your own workflow instead. + [.github/workflows/README.md](../.github/workflows/README.md#what-you-give-up-by-calling-these) + has the trade-off. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..a797bf8 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,14 @@ +# Examples + +Files to copy into your engine's repository. [The quickstart](../docs/QUICKSTART.md) +walks through them. + +| file | copy to | what it is | +| --- | --- | --- | +| `build_at.sh` | `scripts/build_at.sh` | builds the engine at a commit (written for cargo) | +| `strength.yml` | `.github/workflows/strength.yml` | a match you run by hand | +| `sprt.yml` | `.github/workflows/sprt.yml` | a sequential test you run by hand, carried across runs | + +They live here until there's an example repository for them. Nothing in this +repository runs them, but `tests/test_examples.py` checks they call the current +release with inputs it takes. diff --git a/examples/build_at.sh b/examples/build_at.sh new file mode 100755 index 0000000..f856f75 --- /dev/null +++ b/examples/build_at.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MIT +# Copyright (C) 2022-2026 Andrew Wright + +# Build the engine as it was at a commit, to the contract the reusable +# workflows call: +# +# build_at.sh +# +# Written for a Rust engine built with cargo. Change the two lines marked below +# for another build (docs/QUICKSTART.md has them for make and cmake). +set -euo pipefail + +ref=${1:?usage: build_at.sh } +binary=${2:?usage: build_at.sh } + +# Export the commit rather than check it out, so the working tree is left +# alone. `tar -m` stamps the files with the time they're extracted rather than +# the commit's. It doesn't matter while every build starts in a fresh +# directory, but it does once a target directory is shared between builds +# (CARGO_TARGET_DIR, or a cache): cargo would take the sources for older than +# its last build, hand back the binary it already had, and both sides of the +# match would be the same program. +src=$(mktemp -d) +trap 'rm -rf "$src"' EXIT +git archive "$ref" | tar -xm -C "$src" +# git archive leaves out submodules, and Git LFS files come out as pointers. +# If the engine needs either, fetch them into "$src" here. + +mkdir -p "$(dirname "$binary")" +# change these two lines for another build +(cd "$src" && cargo build --release) +cp "${src}/target/release/my-engine" "$binary" diff --git a/examples/sprt.yml b/examples/sprt.yml new file mode 100644 index 0000000..f5656fb --- /dev/null +++ b/examples/sprt.yml @@ -0,0 +1,35 @@ +# A sequential test (SPRT) you run by hand from the Actions tab. It plays up +# to four batches of 500 games and stops as soon as one settles the test. If +# none does, the summary gives you five numbers to paste into prior_pairs next +# time. +name: SPRT + +on: + workflow_dispatch: + inputs: + candidate: + description: Branch, tag, commit or pull request number to test + type: string + baseline: + description: What to measure it against. The last release if empty + type: string + prior_pairs: + description: Pair counts carried from an earlier run. Empty to start a test + type: string + +permissions: + contents: read + +jobs: + test: + uses: aywrite/mache/.github/workflows/strength.yml@v0.6.0 + with: + build: scripts/build_at.sh + candidate: ${{ inputs.candidate }} + baseline: ${{ inputs.baseline }} + sprt: true + elo0: 0 + elo1: 10 + games: 500 + batches: 4 + prior_pairs: ${{ inputs.prior_pairs }} diff --git a/examples/strength.yml b/examples/strength.yml new file mode 100644 index 0000000..50dee17 --- /dev/null +++ b/examples/strength.yml @@ -0,0 +1,30 @@ +# A strength match you run by hand from the Actions tab. Copy it to +# .github/workflows/strength.yml, and build_at.sh to scripts/build_at.sh. +name: Strength + +on: + workflow_dispatch: + inputs: + candidate: + description: Branch, tag, commit or pull request number to test + type: string + baseline: + description: What to measure it against. The last release if empty + type: string + games: + type: number + default: 500 + +# strength.yml asks for no more than this itself. It keeps anything you add +# to this file read-only too. +permissions: + contents: read + +jobs: + match: + uses: aywrite/mache/.github/workflows/strength.yml@v0.6.0 + with: + build: scripts/build_at.sh + candidate: ${{ inputs.candidate }} + baseline: ${{ inputs.baseline }} + games: ${{ inputs.games }} diff --git a/tests/test_examples.py b/tests/test_examples.py new file mode 100644 index 0000000..305f37b --- /dev/null +++ b/tests/test_examples.py @@ -0,0 +1,262 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2022-2026 Andrew Wright + +"""The examples a caller copies, and the quickstart that walks through them. + +Nothing here runs them, since a run needs a repository with an engine in it. +What is checked is what would go wrong quietly once they are copied: a pin at a +release that is not the current one, an input the pinned workflow does not +take, a build script that is not executable or does not do what the contract +says. + +The pins name the version this tree is, like the self pins in the reusable +workflows, so they are moved in the same commit on a release branch. +""" + +import os +import re +import subprocess +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parent.parent +EXAMPLES = ROOT / "examples" +CALLERS = sorted(EXAMPLES.glob("*.yml")) +BUILD = EXAMPLES / "build_at.sh" + +# the documents that show a call, and so name a release +DOCUMENTS = [ + ROOT / "docs" / "QUICKSTART.md", + ROOT / ".github" / "workflows" / "README.md", +] + +VERSION = re.compile(r'^__version__ = "(\d+\.\d+\.\d+)"$', re.MULTILINE) + +# `uses: aywrite/mache/.github/workflows/.yml@` +CALL = re.compile( + r"^(?P *)uses: +aywrite/mache/\.github/workflows/" + r"(?P[a-z-]+\.yml)@(?P\S+)", + re.MULTILINE, +) + + +def version_here(): + found = VERSION.search((ROOT / "mache" / "__init__.py").read_text("utf-8")) + assert found, "mache/__init__.py has no version assignment" + return found.group(1) + + +def release(tag): + """A release tag as numbers to compare, or None for anything else.""" + found = re.fullmatch(r"v(\d+)\.(\d+)\.(\d+)", tag) + return tuple(int(part) for part in found.groups()) if found else None + + +def indent_of(line): + return len(line) - len(line.lstrip()) + + +def keys_at(lines, start, depth): + """The mapping keys exactly `depth` spaces in, from `start` until the block + ends.""" + found = [] + for number in range(start, len(lines)): + line = lines[number] + if not line.strip() or line.lstrip().startswith("#"): + continue + here = indent_of(line) + if here < depth: + break + if here == depth and (key := re.match(r"([^\s:#]+):", line.strip())): + found.append((key.group(1), number)) + return found + + +def workflow_at(workflow, ref): + """The reusable workflow as the caller would get it at `ref`. Off disk + where that tag is this tree's version and not cut yet, which is a release + branch.""" + tagged = subprocess.run( + ["git", "rev-parse", "--verify", f"{ref}^{{commit}}"], + cwd=ROOT, + capture_output=True, + check=False, + ) + if ref == f"v{version_here()}" and tagged.returncode != 0: + # A checkout with no tags at all would land here too, and read the + # tree rather than the release. So this is only a release branch when + # there are tags and the version is newer than all of them. + tags = subprocess.run( + ["git", "tag", "--list", "v[0-9]*"], + cwd=ROOT, + capture_output=True, + text=True, + check=True, + ).stdout.split() + if not tags: + pytest.fail("this checkout has no release tags; fetch them to run this") + newest = max(release(tag) for tag in tags if release(tag)) + assert release(ref) > newest, f"{ref} is not tagged and not new" + return (ROOT / ".github" / "workflows" / workflow).read_text("utf-8") + shown = subprocess.run( + ["git", "show", f"{ref}:.github/workflows/{workflow}"], + cwd=ROOT, + capture_output=True, + text=True, + check=False, + ) + assert shown.returncode == 0, f"{workflow}@{ref} is not readable: {shown.stderr}" + return shown.stdout + + +def declared_inputs(text): + """Every input a reusable workflow takes, and which of them are required.""" + lines = text.splitlines() + start = next(n for n, line in enumerate(lines) if line.strip() == "workflow_call:") + block = next( + n for n in range(start + 1, len(lines)) if lines[n].strip() == "inputs:" + ) + inputs = keys_at(lines, block + 1, indent_of(lines[block]) + 2) + required = set() + for name, number in inputs: + for line in lines[number + 1 :]: + if line.strip() and indent_of(line) <= indent_of(lines[number]): + break + if line.strip() == "required: true": + required.add(name) + return {name for name, _ in inputs}, required + + +def calls(path): + """Each call in a caller, as (workflow, ref, the inputs it passes).""" + text = path.read_text("utf-8") + lines = text.splitlines() + found = [] + for match in CALL.finditer(text): + number = text[: match.start()].count("\n") + depth = len(match.group("indent")) + passed = set() + for offset in range(number + 1, len(lines)): + line = lines[offset] + if line.strip() and indent_of(line) < depth: + break + if indent_of(line) == depth and line.strip() == "with:": + passed = {key for key, _ in keys_at(lines, offset + 1, depth + 2)} + found.append((match.group("workflow"), match.group("ref"), passed)) + return found + + +def test_there_are_examples_to_check(): + # a glob that matched nothing would pass every test below + assert [path.name for path in CALLERS] == ["sprt.yml", "strength.yml"] + + +@pytest.mark.parametrize("path", CALLERS + DOCUMENTS, ids=lambda path: path.name) +def test_every_call_names_the_current_release(path): + found = CALL.findall(path.read_text("utf-8")) + assert found, f"{path.name} shows no call" + for _, workflow, ref in found: + assert ref == f"v{version_here()}", f"{path.name} calls {workflow}@{ref}" + + +def test_the_quickstart_names_no_other_release(): + # the prose as well as the calls, so a sentence about what a release + # takes is looked at again when the pins move. A tag with a suffix, such + # as a pre-release given as an example, is not a release the guide names. + text = (ROOT / "docs" / "QUICKSTART.md").read_text("utf-8") + named = set(re.findall(r"\bv(\d+\.\d+\.\d+)(?![-\w.])", text)) + assert named == {version_here()}, f"the quickstart names {sorted(named)}" + + +def test_the_quickstart_shows_the_example_workflow_as_it_is(): + # the guide shows the whole file, so the two cannot be allowed to differ + text = (ROOT / "docs" / "QUICKSTART.md").read_text("utf-8") + shown = re.findall(r"```yaml\n(.*?)```", text, re.DOTALL) + example = (EXAMPLES / "strength.yml").read_text("utf-8") + body = re.sub(r"\A(#.*\n)+", "", example) + assert body in shown, "the quickstart's copy of strength.yml has drifted" + + +@pytest.mark.parametrize("path", CALLERS, ids=lambda path: path.name) +def test_every_input_passed_is_one_the_pinned_workflow_takes(path): + for workflow, ref, passed in calls(path): + taken, required = declared_inputs(workflow_at(workflow, ref)) + assert passed <= taken, f"{workflow}@{ref} takes no {passed - taken}" + assert required <= passed, f"{path.name} leaves out {required - passed}" + + +@pytest.mark.parametrize("path", CALLERS, ids=lambda path: path.name) +def test_the_caller_keeps_its_token_to_reading(path): + text = path.read_text("utf-8") + assert re.search(r"^permissions:\n contents: read\n", text, re.MULTILINE) + + +def test_the_build_script_is_executable_in_git(): + staged = subprocess.run( + ["git", "ls-files", "--stage", "--", str(BUILD.relative_to(ROOT))], + cwd=ROOT, + capture_output=True, + text=True, + check=True, + ).stdout + if not staged: + pytest.skip("build_at.sh is not in the index yet") + assert staged.startswith("100755"), staged + + +def test_the_build_script_builds_the_commit_it_is_given(tmp_path): + # A repository with two commits and a cargo that copies a file into the + # place a release build would go. The script is handed the first commit + # while the second is checked out, so the binary shows which one it built. + repo = tmp_path / "engine" + repo.mkdir() + # kept away from the developer's own git config, which may sign commits + # or run hooks + env = {**os.environ, "GIT_CONFIG_GLOBAL": os.devnull, "GIT_CONFIG_NOSYSTEM": "1"} + git = ["git", "-c", "user.name=t", "-c", "user.email=t@t", "-C", str(repo)] + subprocess.run([*git, "init", "-q"], check=True, env=env) + for version in ("first", "second"): + (repo / "engine.txt").write_text(f"{version}\n") + subprocess.run([*git, "add", "engine.txt"], check=True, env=env) + # an old date, so an export stamped with the commit's time would show + dated = {**env, "GIT_COMMITTER_DATE": "2001-01-01T00:00:00"} + subprocess.run([*git, "commit", "-qm", version], check=True, env=dated) + first = subprocess.run( + [*git, "rev-parse", "HEAD~1"], + capture_output=True, + text=True, + check=True, + env=env, + ).stdout.strip() + + stubs = tmp_path / "bin" + stubs.mkdir() + stamped = tmp_path / "stamped" + cargo = stubs / "cargo" + cargo.write_text( + "#!/usr/bin/env bash\n" + "set -euo pipefail\n" + # what the export was stamped with, read back below + "python3 -c 'import os, sys; print(int(os.stat(\"engine.txt\").st_mtime))'" + f" > {stamped}\n" + "mkdir -p target/release\n" + "cp engine.txt target/release/my-engine\n" + "chmod +x target/release/my-engine\n" + ) + cargo.chmod(0o755) + + # relative, and from the root of the checkout, as batch.yml calls it + subprocess.run( + ["bash", str(BUILD), first, "tools/new"], + cwd=repo, + env={**env, "PATH": f"{stubs}:{os.environ['PATH']}"}, + check=True, + ) + binary = repo / "tools" / "new" + assert binary.read_text() == "first\n" + assert os.access(binary, os.X_OK) + # stamped when it was extracted, not with the commit's date in 2001 + assert int(stamped.read_text()) > 1_000_000_000 + # the working tree was left as it was + assert (repo / "engine.txt").read_text() == "second\n"