Engine: measurement harness — fixed-node bench + self-play match runner (closes #158) - #159
Merged
Merged
Conversation
…runner (closes #158) There was no way to measure engine strength/speed changes — no bench, perft, self-play, or SPRT anywhere — so tuning was unmeasurable. This adds an internal, non-shipping `EngineLab` package target (not a library product, so it is never linked into the iOS app) plus an `engine-lab` executable. Both only *call* `NegamaxEngine.search`, so the engine's determinism guarantee is untouched. - bench: searches a fixed 20-position suite (openings, tactical middlegames, endgames) under one reproducible limit and prints total nodes, nodes/sec, and a signature — a checksum over each position's best move, score, depth, and node count. Doubles as a determinism regression guard (pinned snapshot in tests) and a speed proxy. - match: engine-vs-engine self-play at fixed nodes/depth (never wall-clock), each opening played twice with colors swapped, reporting W/D/L, score %, and Elo(A - B) with a 95% error margin. Reproducibility: every observable is a pure-integer function of the fixed limits and start position — verified byte-identical across two runs and across debug/release builds. EngineLabTests (run by `swift test`, the ChessKit tests lane) guards the bench signature (stable across runs + pinned snapshot), the Elo/error-margin math, and self-play mechanics: identical engines net exactly even (color-swap fairness), a deeper config outscores a shallower one (the Elo pipeline points the right way), games are reproducible, and every game terminates. Test limits are kept small so the debug suite stays fast. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
Merge request: PR #159 @ 5f3f504
New targets are internal (no new library |
testtest126
enabled auto-merge (squash)
July 12, 2026 17:50
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #158.
What & why
There was no way to measure engine strength/speed changes — no bench, perft, self-play, or SPRT anywhere — so engine tuning was unmeasurable (you couldn't tell an improvement from a regression). This adds the measurement harness that unblocks tuning.
Two tools, in a new internal, non-shipping
EngineLabpackage target (deliberately not a libraryproduct, so it is never linked into the iOS app) plus a thinengine-labexecutable. Both only callNegamaxEngine.search— the engine is untouched, so its determinism guarantee is preserved.1.
bench— determinism guard + speed proxySearches a fixed 20-position suite (openings, tactical middlegames, endgames) under one reproducible limit and prints total nodes, nodes/sec, and a signature (a checksum over each position's best move, score, depth, and node count).
The signature is byte-identical on every machine and across debug/release (engine eval is pure integer) and moves only when search behavior changes — a determinism regression guard, pinned in tests. With a fixed
--depth, the total node count is itself the fingerprint.2.
match— self-play strength measurementEngine-vs-engine at fixed nodes or depth (never wall-clock, so runs reproduce), each opening in a small balanced set played twice with colors swapped. Reports W/D/L, score %, and Elo(A − B) with a 95% error margin.
Two configs differing only in depth/nodes measure how much that extra search is worth in Elo; to compare two versions of the engine, build the tool on each commit and run the same match.
Determinism (the core constraint)
Every observable is a pure-integer function of the fixed limits and start position. Verified byte-identical across two runs and across debug/release builds (same bench signature
0xcd7f…, same match result). Books are off by default (a book's random choice would break reproducibility).Verified locally @ 5f3f504
swift build --package-path ChessKit— all targets compile, incl. the executable, no warnings — EXIT:0swift test --package-path ChessKit— full suite green — EXIT:0--lintand SwiftLint (0.65.0): clean on the new files.EngineLabTests(run by theChessKit testslane) guards: the bench signature (stable across two runs + pinned snapshot), the Elo/error-margin math, and self-play mechanics (identical engines net exactly even; a deeper config outscores a shallower one; games reproduce; every game terminates). Test limits are kept small so the debug suite stays fast.Scope / gates
search.ChessOnline/Messages.swift.