Skip to content

Engine: measurement harness — fixed-node bench + self-play match runner (closes #158) - #159

Merged
testtest126 merged 2 commits into
mainfrom
claude/reverent-shannon-9311f4
Jul 12, 2026
Merged

Engine: measurement harness — fixed-node bench + self-play match runner (closes #158)#159
testtest126 merged 2 commits into
mainfrom
claude/reverent-shannon-9311f4

Conversation

@testtest126

Copy link
Copy Markdown
Owner

Closes #158.

What & why

There was no way to measure engine strength/speed changes — no bench, perft, self-play, or SPRT anywhere — so engine tuning was unmeasurable (you couldn't tell an improvement from a regression). This adds the measurement harness that unblocks tuning.

Two tools, in a new internal, non-shipping EngineLab package target (deliberately not a library product, so it is never linked into the iOS app) plus a thin engine-lab executable. Both only call NegamaxEngine.search — the engine is untouched, so its determinism guarantee is preserved.

1. bench — determinism guard + speed proxy

Searches a fixed 20-position suite (openings, tactical middlegames, endgames) under one reproducible limit and prints total nodes, nodes/sec, and a signature (a checksum over each position's best move, score, depth, and node count).

$ swift run -c release engine-lab bench --nodes 2000
positions: 20   total nodes: 42798   time: 0.33s   speed: 0.13 Mnps
signature: 0xcd7fa918c21eafc2

The signature is byte-identical on every machine and across debug/release (engine eval is pure integer) and moves only when search behavior changes — a determinism regression guard, pinned in tests. With a fixed --depth, the total node count is itself the fingerprint.

2. match — self-play strength measurement

Engine-vs-engine at fixed nodes or depth (never wall-clock, so runs reproduce), each opening in a small balanced set played twice with colors swapped. Reports W/D/L, score %, and Elo(A − B) with a 95% error margin.

$ swift run -c release engine-lab match --depth-a 3 --depth-b 1
A: depth-3    B: depth-1
games: 24  (12 openings × 2 colors)
A results: +15 =9 -0   score: 81.2%
Elo(A - B): +255 ± 120  (95%)

Two configs differing only in depth/nodes measure how much that extra search is worth in Elo; to compare two versions of the engine, build the tool on each commit and run the same match.

Determinism (the core constraint)

Every observable is a pure-integer function of the fixed limits and start position. Verified byte-identical across two runs and across debug/release builds (same bench signature 0xcd7f…, same match result). Books are off by default (a book's random choice would break reproducibility).

Verified locally @ 5f3f504

  • swift build --package-path ChessKit — all targets compile, incl. the executable, no warnings — EXIT:0
  • swift test --package-path ChessKit — full suite green — EXIT:0
  • SwiftFormat (0.62.1) --lint and SwiftLint (0.65.0): clean on the new files.

EngineLabTests (run by the ChessKit tests lane) guards: the bench signature (stable across two runs + pinned snapshot), the Elo/error-margin math, and self-play mechanics (identical engines net exactly even; a deeper config outscores a shallower one; games reproduce; every game terminates). Test limits are kept small so the debug suite stays fast.

Scope / gates

  • Security-sensitive: no — no auth/token/crypto/account-linking/session code; only reads FEN and calls search.
  • App-touching: no — engine package only; iOS lane N/A.
  • Shared files: none — does not touch ChessOnline/Messages.swift.
  • New targets are internal (no new library product), so nothing links into the app or server.

…runner (closes #158)

There was no way to measure engine strength/speed changes — no bench,
perft, self-play, or SPRT anywhere — so tuning was unmeasurable. This adds
an internal, non-shipping `EngineLab` package target (not a library
product, so it is never linked into the iOS app) plus an `engine-lab`
executable. Both only *call* `NegamaxEngine.search`, so the engine's
determinism guarantee is untouched.

- bench: searches a fixed 20-position suite (openings, tactical
  middlegames, endgames) under one reproducible limit and prints total
  nodes, nodes/sec, and a signature — a checksum over each position's best
  move, score, depth, and node count. Doubles as a determinism regression
  guard (pinned snapshot in tests) and a speed proxy.
- match: engine-vs-engine self-play at fixed nodes/depth (never
  wall-clock), each opening played twice with colors swapped, reporting
  W/D/L, score %, and Elo(A - B) with a 95% error margin.

Reproducibility: every observable is a pure-integer function of the fixed
limits and start position — verified byte-identical across two runs and
across debug/release builds.

EngineLabTests (run by `swift test`, the ChessKit tests lane) guards the
bench signature (stable across runs + pinned snapshot), the Elo/error-margin
math, and self-play mechanics: identical engines net exactly even
(color-swap fairness), a deeper config outscores a shallower one (the Elo
pipeline points the right way), games are reproducible, and every game
terminates. Test limits are kept small so the debug suite stays fast.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@testtest126 testtest126 added enhancement New feature or request engine ChessKit engine & rules labels Jul 12, 2026
@testtest126

Copy link
Copy Markdown
Owner Author

Merge request: PR #159 @ 5f3f504

  • Verified locally @ 5f3f504:
    • swift test --package-path ChessKit — 110 tests, 0 failures — EXIT:0
    • swift build --package-path ChessKit — all targets incl. the engine-lab executable, no warnings — EXIT:0
    • SwiftFormat 0.62.1 --lint + SwiftLint 0.65.0 — clean on the new files
    • Determinism cross-check: bench signature 0xcd7fa918c21eafc2 and match results byte-identical across two runs and across debug/release builds
  • Security-sensitive: no (no auth/token/crypto/account-linking/session code — only reads FEN and calls search)
  • App-touching: no (ChessKit engine package only; iOS lane N/A)
  • Shared files: none (does not touch ChessOnline/Messages.swift)
  • Server tests / iOS build not run locally — diff touches neither; Server tests runs as a required check in CI.

New targets are internal (no new library product), so nothing links into the app or server. Ready for the orchestrator to verify green-on-current-merge-state and merge.

@testtest126
testtest126 enabled auto-merge (squash) July 12, 2026 17:50
@testtest126
testtest126 merged commit 0f5e8f7 into main Jul 12, 2026
9 checks passed
@testtest126
testtest126 deleted the claude/reverent-shannon-9311f4 branch July 12, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine ChessKit engine & rules enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Engine: measurement harness — fixed-node bench + self-play match runner (prereq for tuning)

1 participant