From e3fdb9322fcf4ff7b948929d72ec86e160edd8d0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 11:36:15 +0000 Subject: [PATCH 1/2] docs(docs): Lead with what mache is for and what it can measure The README opened on a name and a line about benchmarking, and the case for using it came several paragraphs later. It now starts with what it does and who it suits, and links to two new sections. What hosted runners can measure gives the margins an estimate reaches at a given number of pairs, and how long a normalized sequential test ran in simulation at two sets of bounds. Running a match without CI walks through playing shards with fastchess and pooling them, as it was run while writing it. pyproject.toml gains a summary, keywords and classifiers that say this is for chess engines, so PyPI's search can find it. The README says seven actions where it said six, and a few claims are narrowed to what the code does. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018ccJFsmZ4jAKYmFNgNBh9u --- .github/workflows/README.md | 2 +- .github/workflows/batch.yml | 2 +- .github/workflows/calibrate.yml | 2 +- .github/workflows/strength.yml | 4 +- README.md | 168 +++++++++++++++++++++++++++----- pyproject.toml | 16 ++- 6 files changed, 163 insertions(+), 31 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 392a7df..5f979a2 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -53,7 +53,7 @@ would have been: repository reads a manifest back, it has to read this shape or you have to write your own. -If any of those matters, call the six actions and write the job graph. It is the +If any of those matters, call the actions and write the job graph. It is the right trade when you need a step rather than a string, and the engine this was written for pays it for all three reasons above. diff --git a/.github/workflows/batch.yml b/.github/workflows/batch.yml index d828e0a..1ec90b1 100644 --- a/.github/workflows/batch.yml +++ b/.github/workflows/batch.yml @@ -191,7 +191,7 @@ jobs: book_table: ${{ inputs.book_table }} # Generic, because a reusable workflow cannot be handed a cache action. - # A caller that wants one that understands its toolchain calls the six + # A caller that wants one that understands its toolchain calls the # actions and keeps its own step. - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 if: inputs.cache_paths != '' diff --git a/.github/workflows/calibrate.yml b/.github/workflows/calibrate.yml index 2cef0be..7c29183 100644 --- a/.github/workflows/calibrate.yml +++ b/.github/workflows/calibrate.yml @@ -2,7 +2,7 @@ name: Calibrate # A whole gauntlet against rated opponents, as one call, and the companion to # Strength. The same trade applies: a caller that wants the job graph, its own -# cache action and its own manifest calls the six actions instead. +# cache action and its own manifest calls the actions instead. on: workflow_call: diff --git a/.github/workflows/strength.yml b/.github/workflows/strength.yml index ad40a5f..5d8dd91 100644 --- a/.github/workflows/strength.yml +++ b/.github/workflows/strength.yml @@ -1,7 +1,7 @@ name: Strength # A whole sharded strength match, as one call. A caller that wants the job -# graph, its own cache action and its own manifest calls the six actions +# graph, its own cache action and its own manifest calls the actions # instead and writes about two hundred lines; this is for everyone else, and # costs about twenty. # @@ -91,7 +91,7 @@ on: description: > What to hand actions/cache between runs, one path a line. Empty caches nothing, which is correct and slower. A caller wanting a - toolchain-aware cache action calls the six actions instead: `uses:` + toolchain-aware cache action calls the actions instead: `uses:` cannot be an expression, so no reusable workflow can take one. type: string default: "" diff --git a/README.md b/README.md index 307f140..708f2fd 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,28 @@ # mache -Measure A CHess Engine. +[![Tests](https://github.com/aywrite/mache/actions/workflows/tests.yml/badge.svg)](https://github.com/aywrite/mache/actions/workflows/tests.yml) +[![PyPI](https://img.shields.io/pypi/v/mache)](https://pypi.org/project/mache/) -A collection of tools for measuring and benchmarking chess engines. From mache -(μάχη), Greek for battle. +Chess engine testing on GitHub Actions, with no server to run. + +mache plays a match between two versions of an engine on the GitHub-hosted +runners a repository already has. The match is split across jobs that run at +the same time, and the games are pooled afterwards into one Elo estimate or +one sequential test (SPRT). A repository can run it on its pull requests or +start it by hand, and the result goes in the run's summary. fastchess plays +the games. + +It suits an engine whose changes are still large enough to show within a few +thousand games. A small change can take a great many runs to settle, and +[What hosted runners can measure](#what-hosted-runners-can-measure) says how +many. + +The tools that read the games also work without GitHub Actions, on pgn files +fastchess wrote anywhere. [Running a match without CI](#running-a-match-without-ci) +shows how. + +The name is from mache (μάχη), Greek for battle, and reads as Measure A CHess +Engine. ## Why this exists @@ -17,19 +36,18 @@ The first is that it was not planned. It grew as I wrote [arche](https://github.com/aywrite/arche), my first chess engine, starting as a basic CI job that got out of hand. -The second is that OpenBench needs machines. There is a shared instance a good -many engines develop against, and a dozen or more projects run their own. mache -is for the case where you have neither. It has no instance and no clients, and -it runs on the hosted CI runners a repository already gets, as part of the pull -requests and releases it already runs, so testing an engine costs no machine you -have to own, administer or ask anyone to lend you. Both play their games with -fastchess underneath. +The second is that OpenBench needs machines. Some engines develop against a +shared instance and some projects run their own. mache is for the case where +you have neither. It has no instance and no clients, and it runs on the hosted +CI runners a repository already gets, as part of the pull requests and releases +it already runs, so testing an engine needs no machine of your own. Both play +their games with fastchess underneath. -Hosted runners are the point of mache and also what make it awkward. They are -slow, they are noisy, they are shared, and a job is killed at a time limit often -long before a match worth reading has finished. So a match is split across jobs -that run at once and pooled afterwards, which is most of what the tools below -are for, and why they take the care described further down. +Hosted runners are the point of mache and also what make it awkward. Their +timing varies from one job to the next, and a job is stopped at a time limit, +often long before a match worth reading has finished. So a match is split +across jobs that run at once and pooled afterwards, which is most of what the +tools below are for, and why they take the care described further down. ## What is here @@ -42,7 +60,7 @@ are for, and why they take the care described further down. | `match-terminations` | How the games actually ended | | `book-slice` | Which openings a shard plays, so that no two shards share one | -Six composite actions, which are the parts of a match workflow that are not +Seven composite actions, which are the parts of a match workflow that are not about any one engine. A caller keeps its own jobs, its own matrix and its own toolchain cache, and calls these for the work inside them. @@ -77,6 +95,58 @@ pass a table of its own. There is no build script among them, and `docs/BUILDING-A-REF.md` says why, along with the one trap a build step written for this has to avoid. +## What hosted runners can measure + +A run of `strength.yml` with its defaults plays 500 games over five shards, +which is 250 pairs. A sequential test given `batches: 4` plays up to four such +batches in one run, 1,000 pairs. It stops early if a batch settles the test, +and otherwise hands back the counts for the next run to carry on from. + +For an estimate on its own, the 95% margin in normalized elo, as mache works +it out, depends on the number of pairs and on nothing else: + +| pairs | margin | +| ---: | ---: | +| 250 | ±30 | +| 1,000 | ±15 | +| 4,000 | ±8 | + +The margin in logistic elo also depends on how many of the games are drawn. + +A sequential test runs until the pairs settle it, and the closer its bounds, +the longer that takes. In the simulation below, halving the gap between the +bounds took between three and four times the pairs. It ran the normalized test +with mache's own code, judged every 250 pairs as with the workflow's default +batch, on a drawish distribution of pair scores: + +| bounds (normalized elo) | true difference at a bound | true difference halfway between | +| --- | ---: | ---: | +| [0, 10] | about 4,000 pairs | about 7,000 pairs | +| [0, 5] | about 14,000 pairs | about 26,000 pairs | + +Those are averages, and the spread around them is wide: about one test in ten +ran to nearly twice the average or longer. At 1,000 pairs a run, the averages +come to about four to seven runs for [0, 10] and about fourteen to twenty-six +for [0, 5]. The normalized model is what +`match-estimate --model normalized` and the `sprt_model` input of +`actions/summarise-match` use. The reusable workflows take their bounds in +logistic elo for now, and how long a logistic test runs depends on the draw +rate as well as the bounds. + +How long a batch takes depends on the time control, the engines and the +runner, so it is worth timing one before planning a test around it. A shard +is stopped at `max_match_minutes`, 150 by default, and reports the games it +finished by then. + +Both engines of a shard share one runner, so a slow runner slows both. That +tends to show as noise in the result rather than a lean towards one side, +though an engine whose strength changes a lot with the time it is given can +still be affected more than its opponent. + +GitHub does not charge for its standard hosted runners on public repositories. +A private repository uses the Actions minutes its plan includes. GitHub's +billing documentation has the current terms. + ## Using the tools `pip install mache` gives the four command names used below. Under the @@ -260,6 +330,52 @@ its own games alone would start every batch where the first one started, and the pooled estimate would count those positions twice with nothing failing. A caller that passes neither plays what an unbatched run plays. +## Running a match without CI + +None of the four tools needs GitHub Actions. They read pgn files, or in +`book-slice`'s case a few numbers, so they work on games played anywhere: on +one machine, or on a few machines that each play part of a match. +`pip install mache` gives the command names. fastchess and an opening book are +yours to provide. In a clone of this repository, +`bin/book_table.sh fetch 8moves_v3 .` downloads one of the two books the +actions use and checks it against its recorded hash. + +Each machine plays a shard. `book-slice` says which opening a shard starts at, +so that no two shards play the same one. Every shard asks with the same +numbers and its own `--shard`: + +``` +openings=$(grep -c '^\[Event ' 8moves_v3.pgn) +start=$(book-slice --openings "$openings" --pairs 100 --shards 2 --shard 0 --seed 7) +``` + +Then fastchess plays the shard. `-repeat` makes each round two games on one +opening with the colours reversed, which is what `match-estimate` takes a +round to be. `-rounds` is the number of pairs and has to match `--pairs` +above. `nodes=true` and `timeleft=true` put in the pgn what the report's table +of clocks and nodes is read from; without them that table is all zeros. + +``` +fastchess -engine name=new cmd=./new -engine name=old cmd=./old \ + -each proto=uci tc=10+0.1 \ + -openings file=8moves_v3.pgn format=pgn order=sequential start="$start" \ + -rounds 100 -repeat -concurrency 2 \ + -pgnout file=games.pgn nodes=true timeleft=true +``` + +With each shard's games in a directory of its own, one command pools them. The +directory names become the rows of the shard table: + +``` +match-estimate shard-0/games.pgn shard-1/games.pgn \ + --candidate new --baseline old --tc 10+0.1 +``` + +`--elo0`, `--elo1` and `--prior-pairs` run the sequential test here the same +way they do in a workflow. A test played in several batches passes +`--batches` and `--batch` to `book-slice` as well, so that a later batch does +not replay an earlier one's openings. + ## The part that is not obvious A sharded match is not a long match cut up. Three things have to hold or the @@ -344,20 +460,21 @@ book than on an unbalanced one, and fewer at a long time control than a short one, so two runs that differ in either are not comparable in it. The report also gives the difference in normalized elo, which is what -fastchess prints as nElo and what fishtest states its bounds in. It is the -score's distance from a half divided by the spread of the pairs, scaled so that -a match with no draws reads about the same in both. Because it measures how -clearly the games separate the two sides, it can be compared across books and -time controls, and its margin depends on the number of pairs alone. For the +fastchess prints as nElo. It is the score's distance from a half divided by +the spread of the pairs, scaled so that a small difference in a match with no +draws reads about the same in both. Because it measures how clearly the games +separate the two sides, it compares across books and time controls better +than logistic elo does, and its margin depends on the number of pairs +alone. For the same pairs the figure here is the one fastchess prints. `--json` carries it as `nelo` and `nelo_margin`. The sequential test takes its bounds in either. `--model normalized` reads `--elo0` and `--elo1` as normalized elo, and the default, `--model logistic`, reads them as logistic elo as before. Under the normalized model the number of -games a test needs to settle depends on the bounds and hardly at all on the -book or the time control, so the same bounds cost about the same whichever is -played. A normalized test names itself as `SPRT [0, 5] nElo` in the report, the +games a test needs to settle depends on the bounds, and much less than under +the logistic model on the book or the time control, so the same bounds cost +roughly the same whichever is played. A normalized test names itself as `SPRT [0, 5] nElo` in the report, the line and the trailer, and `--json` carries the model in its `sprt` object. Every batch of one test is judged under the model it started with. The pair @@ -371,7 +488,8 @@ pairs while being that many of its own standard deviations from a half. It does that as a convex fit at each spread and a search over the spread, rather than by the fixed point iteration fastchess uses, which does not converge from every set of counts. Where both converge they agree. Hypotheses are limited to -100 normalized elo either side of nought, far beyond any bound a test uses. +100 normalized elo either side of nought, which is wider than the bounds a test +normally uses. ## Reading a rating estimate diff --git a/pyproject.toml b/pyproject.toml index d528162..f29b0f3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,16 +14,30 @@ name = "mache" # the one in mache/__init__.py, which is where the --json output reads it from, # so there is one version and not two to keep level dynamic = ["version"] -description = "Read a sharded engine match: the pooled estimate and the ccrl fit" +description = "Chess engine testing on GitHub Actions: pool a sharded fastchess match into one Elo estimate or SPRT" readme = "README.md" # the floor the `X | None` annotations set requires-python = ">=3.10" license = "MIT" authors = [{ name = "Andrew Wright" }] dependencies = [] +# What PyPI's search and its project page read. The Python versions are the +# two the tests run on, and no licence classifier is given because the licence +# field above is the expression that says it. +keywords = ["chess", "chess-engine", "elo", "sprt", "fastchess", "github-actions"] +classifiers = [ + "Development Status :: 3 - Alpha", + "Intended Audience :: Developers", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.12", + "Topic :: Games/Entertainment :: Board Games", + "Topic :: Software Development :: Testing", +] [project.urls] Homepage = "https://github.com/aywrite/mache" +Changelog = "https://github.com/aywrite/mache/blob/main/CHANGELOG.md" [project.optional-dependencies] # What the tests need, and nothing else: the tools themselves need nothing, so From c1738150a240b7740723238597fa17f29b886fe6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 11:42:28 +0000 Subject: [PATCH 2/2] docs(docs): Correct and tighten the new README sections An independent read against the code and the test runs found a few things to fix. The clocks table reads as zeros without nodes=true, with a dash for time left, rather than as all zeros. fastchess appends to a pgn that is already there, which the walkthrough now warns about. The [0, 5] row said 14,000 pairs where one of its two results was 13,087. The simulation now says how many tests it ran and what it drew from, and the note on logistic bounds names strength.yml rather than every reusable workflow. The PyPI summary now describes the package, which is the tools and not the actions. The two new sections are linked by full address, since PyPI's copy of the README may not keep heading anchors, and some phrasing is plainer. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018ccJFsmZ4jAKYmFNgNBh9u --- README.md | 87 ++++++++++++++++++++++++++------------------------ pyproject.toml | 2 +- 2 files changed, 47 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 708f2fd..7e0c53c 100644 --- a/README.md +++ b/README.md @@ -5,20 +5,21 @@ Chess engine testing on GitHub Actions, with no server to run. -mache plays a match between two versions of an engine on the GitHub-hosted +mache runs a match between two versions of an engine on the GitHub-hosted runners a repository already has. The match is split across jobs that run at -the same time, and the games are pooled afterwards into one Elo estimate or -one sequential test (SPRT). A repository can run it on its pull requests or -start it by hand, and the result goes in the run's summary. fastchess plays -the games. +the same time, and the games are pooled afterwards into one elo estimate or +one sequential test (SPRT). A repository calls it from a workflow of its own, +on its pull requests or by hand, and the result goes in the run's summary. +fastchess plays the games. It suits an engine whose changes are still large enough to show within a few -thousand games. A small change can take a great many runs to settle, and -[What hosted runners can measure](#what-hosted-runners-can-measure) says how -many. +thousand games. A small change can take many runs to settle, and +[What hosted runners can measure](https://github.com/aywrite/mache#what-hosted-runners-can-measure) +says how many. The tools that read the games also work without GitHub Actions, on pgn files -fastchess wrote anywhere. [Running a match without CI](#running-a-match-without-ci) +fastchess wrote anywhere. +[Running a match without CI](https://github.com/aywrite/mache#running-a-match-without-ci) shows how. The name is from mache (μάχη), Greek for battle, and reads as Measure A CHess @@ -103,7 +104,7 @@ batches in one run, 1,000 pairs. It stops early if a batch settles the test, and otherwise hands back the counts for the next run to carry on from. For an estimate on its own, the 95% margin in normalized elo, as mache works -it out, depends on the number of pairs and on nothing else: +it out, depends only on the number of pairs: | pairs | margin | | ---: | ---: | @@ -111,37 +112,38 @@ it out, depends on the number of pairs and on nothing else: | 1,000 | ±15 | | 4,000 | ±8 | -The margin in logistic elo also depends on how many of the games are drawn. +The margin in logistic elo also depends on how the pairs scored, and so on +how many games were drawn. A sequential test runs until the pairs settle it, and the closer its bounds, the longer that takes. In the simulation below, halving the gap between the -bounds took between three and four times the pairs. It ran the normalized test -with mache's own code, judged every 250 pairs as with the workflow's default -batch, on a drawish distribution of pair scores: +bounds took between three and four times the pairs. It ran 300 tests a row of +the normalized test with mache's own code, judged every 250 pairs as with the +workflow's default batch. The pair scores were drawn from a distribution in +which three pairs in five score one point, shifted to each true difference: | bounds (normalized elo) | true difference at a bound | true difference halfway between | | --- | ---: | ---: | | [0, 10] | about 4,000 pairs | about 7,000 pairs | -| [0, 5] | about 14,000 pairs | about 26,000 pairs | +| [0, 5] | about 13,000 to 14,000 pairs | about 26,000 pairs | Those are averages, and the spread around them is wide: about one test in ten ran to nearly twice the average or longer. At 1,000 pairs a run, the averages -come to about four to seven runs for [0, 10] and about fourteen to twenty-six +come to about four to seven runs for [0, 10] and about thirteen to twenty-six for [0, 5]. The normalized model is what `match-estimate --model normalized` and the `sprt_model` input of -`actions/summarise-match` use. The reusable workflows take their bounds in -logistic elo for now, and how long a logistic test runs depends on the draw -rate as well as the bounds. +`actions/summarise-match` use. `strength.yml` takes its bounds in logistic elo, +so its default of [0, 10] is not the first row above, and how long a logistic +test runs depends on the draw rate as well as the bounds. How long a batch takes depends on the time control, the engines and the runner, so it is worth timing one before planning a test around it. A shard is stopped at `max_match_minutes`, 150 by default, and reports the games it finished by then. -Both engines of a shard share one runner, so a slow runner slows both. That -tends to show as noise in the result rather than a lean towards one side, -though an engine whose strength changes a lot with the time it is given can -still be affected more than its opponent. +Both engines of a shard share one runner, so a slow runner slows both. An +engine whose strength changes a lot with the time it is given loses more to a +slow runner than one whose strength does not. GitHub does not charge for its standard hosted runners on public repositories. A private repository uses the Actions minutes its plan includes. GitHub's @@ -349,11 +351,11 @@ openings=$(grep -c '^\[Event ' 8moves_v3.pgn) start=$(book-slice --openings "$openings" --pairs 100 --shards 2 --shard 0 --seed 7) ``` -Then fastchess plays the shard. `-repeat` makes each round two games on one -opening with the colours reversed, which is what `match-estimate` takes a -round to be. `-rounds` is the number of pairs and has to match `--pairs` -above. `nodes=true` and `timeleft=true` put in the pgn what the report's table -of clocks and nodes is read from; without them that table is all zeros. +Then fastchess plays the shard. `-repeat` pairs the games as `match-estimate` +expects. `-rounds` is the number of pairs and has to match `--pairs` above. +`nodes=true` is what the report's table of clocks and nodes counts moves by, +and `timeleft=true` gives its last column. Without them the table reads as +zeros. ``` fastchess -engine name=new cmd=./new -engine name=old cmd=./old \ @@ -363,6 +365,9 @@ fastchess -engine name=new cmd=./new -engine name=old cmd=./old \ -pgnout file=games.pgn nodes=true timeleft=true ``` +fastchess adds to a `games.pgn` that is already there, so start each shard in +an empty directory. A shard played twice into one file would be counted twice. + With each shard's games in a directory of its own, one command pools them. The directory names become the rows of the shard table: @@ -371,8 +376,8 @@ match-estimate shard-0/games.pgn shard-1/games.pgn \ --candidate new --baseline old --tc 10+0.1 ``` -`--elo0`, `--elo1` and `--prior-pairs` run the sequential test here the same -way they do in a workflow. A test played in several batches passes +`--elo0`, `--elo1`, `--prior-pairs` and `--model` run the sequential test here +the same way they do in a workflow. A test played in several batches passes `--batches` and `--batch` to `book-slice` as well, so that a later batch does not replay an earlier one's openings. @@ -459,23 +464,23 @@ how often the games are drawn. The same change reads as fewer elo on a balanced book than on an unbalanced one, and fewer at a long time control than a short one, so two runs that differ in either are not comparable in it. -The report also gives the difference in normalized elo, which is what -fastchess prints as nElo. It is the score's distance from a half divided by -the spread of the pairs, scaled so that a small difference in a match with no -draws reads about the same in both. Because it measures how clearly the games -separate the two sides, it compares across books and time controls better -than logistic elo does, and its margin depends on the number of pairs -alone. For the -same pairs the figure here is the one fastchess prints. `--json` carries it as -`nelo` and `nelo_margin`. +The report also gives the difference in normalized elo, which is what fastchess +prints as nElo. It is the score's distance from a half divided by the spread of +the pairs, scaled so that a small difference in a match with no draws reads +about the same in both. Because it measures how clearly the games separate the +two sides, it compares across books and time controls better than logistic elo +does, and its margin depends on the number of pairs alone. For the same pairs +the figure here is the one fastchess prints. `--json` carries it as `nelo` and +`nelo_margin`. The sequential test takes its bounds in either. `--model normalized` reads `--elo0` and `--elo1` as normalized elo, and the default, `--model logistic`, reads them as logistic elo as before. Under the normalized model the number of games a test needs to settle depends on the bounds, and much less than under the logistic model on the book or the time control, so the same bounds cost -roughly the same whichever is played. A normalized test names itself as `SPRT [0, 5] nElo` in the report, the -line and the trailer, and `--json` carries the model in its `sprt` object. +roughly the same whichever is played. A normalized test names itself as +`SPRT [0, 5] nElo` in the report, the line and the trailer, and `--json` +carries the model in its `sprt` object. Every batch of one test is judged under the model it started with. The pair counts carried between batches are the same under either, so nothing stops a diff --git a/pyproject.toml b/pyproject.toml index f29b0f3..98a99da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,7 +14,7 @@ name = "mache" # the one in mache/__init__.py, which is where the --json output reads it from, # so there is one version and not two to keep level dynamic = ["version"] -description = "Chess engine testing on GitHub Actions: pool a sharded fastchess match into one Elo estimate or SPRT" +description = "Pool a sharded fastchess match into one elo estimate or sequential test, and place a chess engine on a rating list" readme = "README.md" # the floor the `X | None` annotations set requires-python = ">=3.10"