diff --git a/docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md b/docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md new file mode 100644 index 00000000..9732210d --- /dev/null +++ b/docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md @@ -0,0 +1,159 @@ +# Targeted sampling, PR 2: the weight column — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** every memory row carries a float32 `weight`; the refit's loss and the oracle weigh a row by `t × weight`; weight ≡ 1 reproduces today's run bit for bit, and a memory file written before the column existed loads with weights of one. + +**Architecture:** the column is a fourth array in `Reservoir`, riding the same slot as its row through Algorithm R's eviction, save and load; `contents()` grows from three arrays to four and that tuple flows unchanged through `step` into `Learner.fit`. The loss multiplies `t` by `weight` before the batch-mean normalisation; the oracle multiplies each sample by its weight before LCFR's per-t′ bank. Nothing consumes weights other than one yet: `collect.rows_of` keeps producing unweighted rows, `merge` without a weight stamps ones, and PR 4's boost stage will be the first caller to pass the column. + +**Tech Stack:** Python 3.12+, NumPy; torch only inside `fit.py`, `policy.py` bodies and the torch-gated tests. + +**Spec:** the targeted-sampling plan, outside the repo, `/Users/alec/Desktop/Claude/poker/targeted-sampling-plan/targeted-sampling-plan.md` — §3 (why an exact row's weight is `(1 − α) · K · arrival · t` and why the expected weight must be unchanged) and the appendix sections "PR 2 — weight column" (the contract) and "PR 4 — the boost stage" (the consumer). + +## Global Constraints + +- The oracle pin `tests/deepcfr/test_train.py::test_the_oracle_loop_is_lcfr_bit_for_bit` passes unchanged. The oracle banks `Σ t′ · (w · r)` per t′ in increasing order with `np.add.at` in memory order; with w ≡ 1 the multiplication is exact and the bits do not move. +- The loss is `(t · w) / mean(t · w)` per row; with w ≡ 1 this is `t / mean(t)` bit for bit (`t · 1.0 = t` in float32). The weight is a sample weight, never a divisor of the target: `pot_scaled` composes with it untouched. +- `weight ≥ 0` and finite; a row of weight 0 is inert in the loss and in the oracle's bank; a weight of 2 equals the same row offered twice (oracle). Rows stay in offer order. +- A reservoir file with no `weight` key loads with ones (a saved run resumes as the run it was); a file with a `weight` of the wrong length or dtype is refused by name. +- No new `DeepConfig` field: nothing to add to `ADDED_LATER` or `SAMPLE_FIELDS`. The resume state's file set does not change; the weight lives inside each memory's `.npz`. +- torch stays optional; the default suite passes without it. `uv run pytest` is the gate (CI runs no tests). +- Every prose claim that says a row is 24 bytes, a 50M-row memory is 1.2 GB, or the row is `(row, t, target)` is corrected in the same change: `reservoir.py`, `fit.py`, `train.py`, `policy.py`, `deepcfr/__init__.py`, `deepcfr/AGENTS.md`, `scripts/deepcfr/modal_rung4.py:155`, `tests/deepcfr/test_reservoir.py`. +- Commit subjects `feat(rung4): `; branch from PR 1's head (`23cd464`), never rebased onto main. No `git stash`, no Modal. + +## Review Focus + +1. A minibatch whose every row has weight 0 (small memory, α = 0 in PR 4): the loss must be finite, not 0/0 = NaN that ends the fit as "diverged" — pinned in Task 2 (`test_a_batch_of_weightless_rows_has_a_finite_loss`). +2. A memory whose every row has weight 0: refused by name before the fit, like the empty memory — Task 2 (`test_fit_refuses_bad_weights`). +3. A weight landing in the wrong slot during eviction (the column assigned with a different index than the row's) — Task 1 (`test_a_weight_rides_its_row_through_eviction`). +4. A `weight` array saved as float64 or of the wrong length being poured into the float32 column in silence — Task 1 (`test_load_refuses_a_weight_of_the_wrong_length_or_type`). +5. The grader's Π fit and the sweep script unpacking three arrays from a `contents()` that now returns four — Task 2 (`fit_policy` takes the fourth positionally; `pi_loss_sweep.py` unpacks four). + +--- + +### Task 1: the column in the reservoir + +**Files:** +- Modify: `src/drawmaha_solver/deepcfr/reservoir.py` (module docstring, `_FILE_KEYS`, `Reservoir.__init__`, `merge`, `contents`, `save`, `load`, `_validate_offer`) +- Modify: `src/drawmaha_solver/deepcfr/AGENTS.md:72-73`, `src/drawmaha_solver/deepcfr/__init__.py:16-18`, `scripts/deepcfr/modal_rung4.py:155` +- Test: `tests/deepcfr/test_reservoir.py` + +**Interfaces:** +- Produces: `Reservoir.merge(rows, targets, *, t, rng, weight=None)` — `weight` (n,) float, finite, ≥ 0; `None` stamps every row 1.0. `Reservoir.contents() -> (rows int32, t int32, targets float32 (n, 4), weight float32 (n,))`, read-only views. `save` writes a `weight` key; `load` reads it when present, else ones; `_WEIGHT_KEY = "weight"` beside `_FILE_KEYS`, which stays the required set. + +- [ ] **Step 1: Write the failing tests** in `tests/deepcfr/test_reservoir.py` + +```python +def test_rows_offered_without_a_weight_weigh_one(): + rows, t, kept, weight = filled_memory().contents() + assert weight.dtype == np.float32 and weight.tolist() == [1.0] * 5 + +def test_a_weight_rides_its_row_through_eviction(): + memory = Reservoir(capacity=5) + memory.merge(np.arange(3), targets(3), t=1, rng=np.random.default_rng(0), weight=np.array([0.5, 1.0, 1.5])) + memory.merge(np.arange(4) + 10, targets(4), t=2, rng=FixedDraws([0, 7]), weight=np.array([2.0, 3.0, 4.0, 5.0])) + rows, _, _, weight = memory.contents() + assert rows.tolist() == [12, 1, 2, 10, 11] + assert weight.tolist() == [4.0, 1.0, 1.5, 2.0, 3.0] + +def test_merge_refuses_bad_weights(): + # negative, NaN, wrong length, 2-D: each a ValueError matching "weight" + +def test_a_file_written_before_the_weight_existed_loads_with_ones(tmp_path): + # np.savez with rows/t/targets/offered/capacity only → load → weight == ones(size), and the next merge still works + +def test_load_refuses_a_weight_of_the_wrong_length_or_type(tmp_path): + # weight of length size − 1 → ValueError matching the file name; weight float64 → "weight must be float32" +``` + +Extend `test_contents_are_read_only_and_feed_the_fit` (four views; `weight` float32 `(5,)`, read-only) and `test_never_exceeds_capacity_and_counts_every_offer` (weights given as `rows % 100` so the fourth column moves with the other three). The round trip in `test_save_is_atomic_and_load_refuses_a_foreign_file` already zips `contents()` strictly. + +- [ ] **Step 2: Run them to make sure they fail**: `uv run pytest tests/deepcfr/test_reservoir.py -q -p no:cacheprovider` — the new tests fail on `TypeError` (unexpected `weight`) and on unpacking three values into four. + +- [ ] **Step 3: Implement** — a fourth `np.zeros(capacity, float32)` array; `_validate_offer` grows a `weight` parameter and returns it as float32 (ones when `None`); `merge` assigns `self._weight[slot[winners]] = weight[winners]`; `contents` returns four views; `save` writes `weight=`; `load` reads `saved[_WEIGHT_KEY]` when present (shape `(size,)`, dtype float32, both refused by name) else `np.ones(size, float32)`. Docstrings: the row is `(row, t, target, weight)`, 28 bytes, a 50M-row memory 1.4 GB; the weight's meaning (how many sampled rows this row stands for; 1 for every row the walk offers, PR 4's exact rows carry `(1 − α) · K · arrival`); why a missing key reads as one. + +- [ ] **Step 4: Run the reservoir tests and the default rung-4 suite**: `uv run pytest tests/deepcfr -q -p no:cacheprovider` — `test_train.py` fails at `Spy.fit` and `test_fit_receives_the_memory_as_it_stands_after_the_merge` (four arrays into a three-way zip) until Task 3; everything else passes. + +- [ ] **Step 5: Commit** `feat(rung4): every memory row carries a float32 weight, one unless a caller says otherwise; a file without the column loads as ones` + +### Task 2: the loss weighs a row by t × weight + +**Files:** +- Modify: `src/drawmaha_solver/deepcfr/fit.py` (module docstring, `batch_loss`, `_weighted_loss`, `fit_from_scratch`, `_fit_on_device`) +- Modify: `src/drawmaha_solver/deepcfr/policy.py` (`fit_policy`, its docstring at line 24), `scripts/deepcfr/pi_loss_sweep.py:225` +- Test: `tests/deepcfr/test_fit.py` (torch-gated) + +**Interfaces:** +- Consumes: `Reservoir.contents()` four-tuple (Task 1). +- Produces: `batch_loss(net, rows, t, targets, weight=None) -> Tensor`; `_weighted_loss(outputs, gather, targets, t, weight) -> Tensor` with `scale = t * weight; (scale / scale.mean().clamp(min=tiny) * error).mean()`; `fit_from_scratch(rows, t, targets, weight=None, *, seed, config)` refusing a `weight` that is not 1-D, not the memory's length, non-finite, negative, or with no positive entry ("no row carries weight: nothing to fit"); `_fit_on_device(rows, t, targets, weight, folded, config, *, device)`; `fit_policy(rows, t, targets, weight=None, *, seed, iteration, config)`. + +- [ ] **Step 1: Write the failing tests** in `tests/deepcfr/test_fit.py` + +```python +def test_weight_one_is_the_unweighted_fit_bit_for_bit(): + rows, t, targets = real_rows(300, seed=4) + plain = fit_from_scratch(rows, t, targets, seed=(0, 1, 0), config=SMALL) + ones = fit_from_scratch(rows, t, targets, np.ones(300, np.float32), seed=(0, 1, 0), config=SMALL) + for a, b in zip(plain.parameters(), ones.parameters(), strict=True): + assert torch.equal(a, b) + +def test_a_row_of_weight_zero_is_inert(): + # one row twice: +1 at weight 0, −1 at weight 1, both t = 1 → prediction −1 within 0.05 + +def test_the_loss_weighs_a_row_by_t_times_weight(): + # one row twice: +1 at (t = 1, w = 3) and −1 at (t = 3, w = 1): (3·1 − 3·1)/6 = 0; swap to w = (1, 1): (1 − 3)/4 = −0.5 + +def test_a_batch_of_weightless_rows_has_a_finite_loss(): + # batch_loss with weight = zeros → 0.0, finite, backward finite + +def test_fit_refuses_bad_weights(): + # negative → "weight"; NaN → "finite"; length n − 1 → "length"; all zero → "nothing to fit"; 2-D → "1-D" +``` + +Pass `weight=np.ones(...)` through `test_the_device_loop_on_the_cpu_is_the_cpu_fit_bit_for_bit`'s `_fit_on_device` call. + +- [ ] **Step 2: Run to make sure they fail** — torch is absent on this Mac, so run under the `deep` extra once: `uv run --extra deep pytest tests/deepcfr/test_fit.py -q -p no:cacheprovider`. Expected: the new tests fail on `TypeError`. + +- [ ] **Step 3: Implement** the signatures above. In `fit_from_scratch`, `weight = np.ones(len(rows), np.float32)` when `None`, else `np.asarray(weight, dtype=np.float32)` validated after `t`. In `NetLearner` (Task 3) and the cpu loop, index `weight[picked]` beside `t[picked]`. Docstring: the loss paragraph says `t · weight`; the device paragraph counts the weight in the copy. + +- [ ] **Step 4: Run** `uv run --extra deep pytest tests/deepcfr/test_fit.py tests/deepcfr/test_policy.py -q -p no:cacheprovider` → PASS; then `uv sync` to drop torch again and `uv run pytest tests/deepcfr/test_fit.py -q` → skipped. + +- [ ] **Step 5: Commit** `feat(rung4): the refit weighs a row by t × weight; weight one is today's loss bit for bit and a row of weight zero is inert` + +### Task 3: the learner seam and the oracle + +**Files:** +- Modify: `src/drawmaha_solver/deepcfr/train.py` (module docstring lines 15, 36–37, 55–65; `Learner.fit`; `step`'s docstring; `NetLearner.fit`; `OracleModel`; `OracleLearner.fit/write/save/load`) +- Modify: `src/drawmaha_solver/deepcfr/AGENTS.md` (the oracle-pin rule names the weight) +- Test: `tests/deepcfr/test_train.py`, `tests/deepcfr/test_train_torch.py` + +**Interfaces:** +- Consumes: Task 1's `contents()`, Task 2's `fit_from_scratch` and `batch_loss`. +- Produces: `Learner.fit(rows, t, targets, weight, *, seed) -> Fitted` (weight required at the seam, so no learner can ignore the column by accident); `OracleModel(rows, t, targets, weight)` with `weight: float32 (n,)`; `OracleLearner.write` banks `summed[touched] += int(t) * totals` where `totals` sums `values * repeat(weight, width)` in float64. + +- [ ] **Step 1: Write the failing tests** in `tests/deepcfr/test_train.py` + +```python +def test_the_oracle_weighs_a_row_by_its_weight_and_a_weight_of_two_is_the_row_twice(): + # three OracleModels over one real row r = (0.5, −0.25, 0, 0) at t = 3: + # twice at weight 1 == once at weight 2 (array_equal) + # once at weight 1 + once at weight 0 == once at weight 1 (array_equal) +def test_oracle_models_round_trip # extend names to ("rows", "t", "targets", "weight") +def test_an_oracle_model_saved_without_a_weight_loads_with_ones(tmp_path) +``` + +Update `Spy.fit(self, rows, t, targets, weight, *, seed)` to record four arrays; `test_fit_receives_the_memory_as_it_stands_after_the_merge` zips four. In `test_train_torch.py` pass `np.ones(n, np.float32)` as the fourth positional at lines 110, 112, 149, 154, 159 and in the `spy` there (`spy(rows, t, targets, weight=None, *, seed, config)`). + +- [ ] **Step 2: Run to make sure they fail**: `uv run pytest tests/deepcfr/test_train.py -q -p no:cacheprovider -x`. + +- [ ] **Step 3: Implement.** `step` is untouched but for its docstring (natural rows weigh one; PR 4's boost stage passes the column). `NetLearner.fit` threads `weight` into `fit_from_scratch` and the held `batch_loss`. The oracle: `values = model.targets[at][legal].astype(np.float64) * np.repeat(model.weight[at].astype(np.float64), width)`. + +- [ ] **Step 4: Run** `uv run pytest tests/deepcfr -q -p no:cacheprovider` → all pass, the oracle pin among them; then `uv run --extra deep pytest tests/deepcfr -q -p no:cacheprovider` for the torch half, then `uv sync`. + +- [ ] **Step 5: Commit** `feat(rung4): the learner seam and the oracle take the weight; the oracle banks Σ t′ · w · r in LCFR's order, so weight one keeps the pin bit for bit` + +### Task 4: the whole suite, a fresh-context review, the PR + +- [ ] `uv run pytest -q -p no:cacheprovider` (about seven minutes; the LCFR 20M run is live, so `nice`). +- [ ] Dispatch a read-only reviewer subagent over `git diff origin/main...` with the plan's Global Constraints and Review Focus; fix what it finds. +- [ ] Push, open the PR against `main` with the house body (what it is, why the weight is where it is, the tests, rulings against the appendix, not in this PR), then watch `claude-review` and answer or fix every finding until none remain. diff --git a/docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md b/docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md new file mode 100644 index 00000000..40504a61 --- /dev/null +++ b/docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md @@ -0,0 +1,95 @@ +# Boost target: modular sampling from a high-noise row — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Turn PR #89's exact target into a sampling framework. At a targeted row, it runs M extra play-outs from that row. Each source of randomness below the row is sampled k-of-n or enumerated, chosen per call. The result is unbiased for the regret the walk's samples estimate. Exact enumeration remains as the all-enumerate special case and the test oracle. + +**Architecture:** One value type, `Sampling`, names a count per source: opponent range, board cards, opponent draws, hero draws, opponent actions, and the number of play-outs. The existing downward recursion (`_value`, `_action_values`, `_push`) takes a private `_Draws` object. That object either returns everything (exact) or a random subset with its unbiasing scale. The upward back-solve (`_reach`) and the arrival stay exact always: the arrival is the cheap half, and an exact denominator under a sampled numerator keeps the ratio unbiased. Randomness is one `default_rng([seed, t, seat, row, TAG])` per row, so a row's result never depends on how rows are batched or split across workers. + +**Tech Stack:** Python 3.12, NumPy 2.5 (`Generator.multinomial` with 3-D `pvals`), pytest, `uv`. + +**Spec:** the chat design agreed 2026-11-06 (Alec): "sample extra starting from node x because node x is high variance … modular … board, opponent draws, hero draws, opp actions … unbiased". This builds on `~/Desktop/Claude/poker/targeted-sampling-plan/targeted-sampling-plan.md` §3 (row weight K · arrival · t, unchanged). + +## Global Constraints + +- The module imports `rules/` and the packed table only: never `exploitability.py`, `deepcfr`, or torch. The static import test stays. +- `Sampling` fields: `playouts: int = 1`, `opponent_range: int | None = None`, `board_cards: int | None = 1`, `opponent_draws: int | None = 1`, `hero_draws: int | None = 1`, `opponent_actions: int | None = 1`. `None` means enumerate all. `EXACT = Sampling(1, None, None, None, None, None)`. +- Chance sources (board, opponent draws, hero draws) sample k of n **without** replacement with scale n/k; k ≥ n is enumeration. +- Opponent actions are k i.i.d. draws from σ per opponent state, weighted count/k; `None` means split by σ. +- Opponent range is k i.i.d. hands per entry, drawn from that entry's compatible weight, each carrying W/k; `None` means the whole range. +- The arrival is always exact; `targets()` returns the same arrival as `arrivals()`, bit for bit. +- `targets(..., sampling=EXACT)` equals the pre-change output bit for bit. +- Stream: `np.random.default_rng([seed, t, seat, row, _STREAM_TAG])`; a non-EXACT `sampling` without `seed` and `t` is refused by name. +- Rename: `solver/exact_target.py` → `solver/boost_target.py`, `ExactTargets` → `BoostTargets`, `tests/minidrawmaha/test_exact_target.py` → `test_boost_target.py`; update `minidrawmaha/AGENTS.md`. +- The default suite stays torch-free and under its current time budget. Sampled unbiasedness tests use the restricted `SMALL`/`WIDE` deal sets. + +## Review Focus + +1. A sampled dimension that never actually samples, because the row has nothing of that kind below it. Every unbiasedness case must also assert that the sampled spread is > 0. +2. Hero-draw sampling must not leak into the hero's own **actions**. Hero actions are always enumerated under the hero's mix; only the replacement card is sampled. +3. Pruning a zero-weight opponent branch must not change the EXACT path. This is pinned by the bit-for-bit test. +4. `opponent_range = k` at a row whose compatible weight is zero (an unreachable row) must return zero regret, not NaN or a sampling error. +5. The same row requested twice in one call, or in two calls with rows in a different order, must give identical sampled results. + +--- + +### Task 1: rename and the `Sampling` value type + +**Files:** +- Rename: `src/drawmaha_solver/minidrawmaha/solver/exact_target.py` → `boost_target.py`; `tests/minidrawmaha/test_exact_target.py` → `test_boost_target.py` +- Modify: `src/drawmaha_solver/minidrawmaha/AGENTS.md` (the `exact_target` line) + +**Interfaces — Produces:** `Sampling` (frozen dataclass, fields above), `EXACT`, `BoostTargets` (the renamed class). `BoostTargets.targets(sigma, *, seat, rows, sampling: Sampling, seed: int | None = None, t: int | None = None) -> tuple[np.ndarray, np.ndarray]`. `sampling` is required, so every caller states its choice. + +- [ ] Step 1: Snapshot today's outputs (not committed). Save `targets` on `random_table(9)` for 40 rows spread over round-1, draw and round-2 points, both seats, to `.context/exact_snapshot.npz`. +- [ ] Step 2: `git mv` both files, update imports and the AGENTS.md line, and add `sampling=EXACT` at every existing test call site. Run `uv run pytest tests/minidrawmaha/test_boost_target.py tests/test_agent_docs.py`: PASS. +- [ ] Step 3: Failing test `test_sampling_refuses_counts_that_are_not_positive_ints`: `Sampling(board_cards=0)`, `Sampling(hero_draws=-1)`, `Sampling(opponent_actions=True)`, `Sampling(playouts=0)` and `Sampling(playouts=2, …all None)` each raise `ValueError`, and the message names the field. +- [ ] Step 4: Failing test `test_a_sampled_call_without_seed_and_t_is_refused`. +- [ ] Step 5: Implement `Sampling.__post_init__` and the refusal in `targets`, then run the tests: PASS. Commit `feat(rung4): the boost target names what it samples — one Sampling value, EXACT the oracle`. + +### Task 2: the `_Draws` seam and the exact path through it + +**Files:** `boost_target.py` (`_value`, `_action_values`, `_push`, `targets`) + +**Interfaces — Produces:** `_Draws(sampling, rng)` with: +- `board(kids: (nb,14)) -> (kids', scale)` +- `replacements(n_rows, fan=12) -> (pick | None, scale)`, used by both hero and opponent draws with their own field +- `actions(pvals: (nb,S,w)) -> weights (nb,S,w)` +- `range(u, mask, k) -> u'` + +On `EXACT`, every method is the identity, with no RNG and no array copy. + +- [ ] Step 1: Failing test `test_exact_sampling_reproduces_the_snapshot_bit_for_bit`. This is a temporary local test reading `.context/exact_snapshot.npz`; it's deleted before the commit, and its committed successor is Step 3 of this task. +- [ ] Step 2: Thread `draws: _Draws` through `_action_values` and `_value`, and give `_push` an optional `pick`. Skip an opponent child whose pushed weight has no positive entry (it returns zeros). Run: PASS bit for bit. +- [ ] Step 3: Committed test `test_a_count_at_or_above_the_fan_is_enumeration`: `Sampling(board_cards=14, opponent_draws=12, hero_draws=12, opponent_actions=None, opponent_range=None)` equals `EXACT` with `np.array_equal`, on a draw row and a round-1 row. +- [ ] Step 4: Commit `feat(rung4): the play-out takes its draws from one seam; exact is the seam returning everything, bit for bit`. + +### Task 3: the five samplers, each unbiased and non-vacuous + +**Files:** `boost_target.py` (`_Draws` bodies, the playouts loop, the per-row stream), `test_boost_target.py` + +- [ ] Step 1: Failing test `test_each_sampled_source_is_unbiased_and_actually_samples`, parametrized over `opponent_range=1`, `board_cards=1`, `opponent_draws=1`, `hero_draws=1`, `opponent_actions=1`, and `Sampling()` (all at once). Each case uses a row where that source is live, under `build(holes=WIDE)`, a random positive table, 400 seeds (t = seed). Assertions: + - for every action column, |mean − exact| < 4 · s.e. + - `assert_pooled_fair` on the pooled z + - the per-column spread is > 0 for at least one column (Review Focus 1) + - `arrival` equals `arrivals()` exactly +- [ ] Step 2: Implement the samplers: + - **Board:** `rng.permuted` over the 14 children per batch row, take the first k, scale 14/k into `u`. + - **Replacements:** k of 12 per (row, hole) or (row, hero state), via argsort of uniforms, scale 12/k. + - **Actions:** `rng.multinomial(k, pvals) / k`; rows of invalid states get uniform `pvals`, and their weight is already zero. + - **Range:** per entry, mask the states that clash with the hero's cards, draw k states ∝ `u·mask` with `rng.choice`, and put W/k on each draw (with multiplicity); W = 0 returns zeros. + - **Play-outs:** loop M times per row, averaging the numerator; divide once by the exact arrival. + + Run: PASS. +- [ ] Step 3: Failing test, then passing: `test_a_rows_sample_depends_on_seed_t_seat_row_only`. The same rows in reversed order, split over two calls, and with a duplicate give equal results; t + 1 gives a different result (Review Focus 5). +- [ ] Step 4: Failing test, then passing: `test_more_playouts_shrink_the_spread`. Across 200 seeds, the variance at `playouts=8` is within [1/16, 1/4] of the variance at `playouts=1`. +- [ ] Step 5: Failing test, then passing: `test_an_unreachable_row_is_zero_under_any_sampling`, with `opponent_range=1` (Review Focus 4). +- [ ] Step 6: Commit `feat(rung4): any source below a boosted row can be sampled k of n — each unbiased against the exact sum, on a per-row stream`. + +### Task 4: cost table, docstring, PR, and PR 2 + +- [ ] Step 1: Extend the benchmark test to print ms per row for `EXACT` vs `Sampling()` at round-1, draw and round-2 rows. Keep the existing round-2 ceiling, applied to `EXACT`. +- [ ] Step 2: Rewrite the module docstring: what is sampled, why the arrival stays exact (ratio bias), why k-of-n without replacement for chance and i.i.d. for σ, the stream, and the measured costs. Every number in it must be recomputed by a test or printed by the benchmark. +- [ ] Step 3: `uv run pytest`, the full default suite: PASS. Push to `AlecROndo/canary-handoff-pr-1-build`, retitle #89 and rewrite its body. +- [ ] Step 4: PR 2: in a temporary detached worktree on `origin/AlecROndo/pr-2-handoff-prompt`, `git merge origin/AlecROndo/canary-handoff-pr-1-build` (merge, never rebase), run `uv run pytest`, and push `HEAD:AlecROndo/pr-2-handoff-prompt`. Workspace `bariloche`'s local branch then needs a `git pull`; say so. +- [ ] Step 5: Drive #89 and #91 to merge-ready: checks green, `claude-review` findings dispositioned. diff --git a/scripts/deepcfr/modal_rung4.py b/scripts/deepcfr/modal_rung4.py index cb671cae..f9f29ae6 100644 --- a/scripts/deepcfr/modal_rung4.py +++ b/scripts/deepcfr/modal_rung4.py @@ -152,7 +152,7 @@ ) TRAINER = {"image": image, "cpu": WORKERS + 1, "memory": MEMORY_GIB * 1024} # The grader holds the compiled game (1.5 GB), a table, a policy memory of -# up to 1.2 GB and Π's fit; eight threads for that fit. +# up to 1.4 GB and Π's fit; eight threads for that fit. GRADER = {"image": image, "cpu": GRADER_CPU, "memory": GRADER_GIB * 1024} def _run_dir(name: str) -> Path: diff --git a/scripts/deepcfr/pi_loss_sweep.py b/scripts/deepcfr/pi_loss_sweep.py index b59aef1b..cf35b090 100644 --- a/scripts/deepcfr/pi_loss_sweep.py +++ b/scripts/deepcfr/pi_loss_sweep.py @@ -222,7 +222,7 @@ def main() -> None: continue mark = mark_of(arm) if loaded is None or loaded[0] != mark: - rows, t, targets = Reservoir.load(run_dir / "marks" / f"t{mark:05d}" / f"policy-t{mark:05d}.npz").contents() + rows, t, targets, _ = Reservoir.load(run_dir / "marks" / f"t{mark:05d}" / f"policy-t{mark:05d}.npz").contents() exact = load_strategy(run_dir / "frozen" / f"t{mark:05d}-exact.npz") loaded = (mark,) print(f"mark t={mark}: memory {len(rows):,} rows", flush=True) diff --git a/src/drawmaha_solver/deepcfr/AGENTS.md b/src/drawmaha_solver/deepcfr/AGENTS.md index 0d6cb5bd..1edb9b30 100644 --- a/src/drawmaha_solver/deepcfr/AGENTS.md +++ b/src/drawmaha_solver/deepcfr/AGENTS.md @@ -22,9 +22,10 @@ itself is complete; what remains is the paid run and its grades. game sit behind `MINIDRAWMAHA_FULL_DEEP=1`. - **The oracle pin is the correctness gate.** With `OracleLearner` the loop must reproduce rung 3's LCFR table bit for bit over three iterations. A - change to the loop's order, to the float32 cast in `collect.rows_of`, or to - `regret_rules.bank_rows` breaks that pin. If it breaks, the loop is wrong, - not the test. + change to the loop's order, to the float32 cast in `collect.rows_of`, to + `regret_rules.bank_rows`, or to where the row weight multiplies (it scales + each sample in float64 before LCFR's per-t′ sum, so weight one is exact) + breaks that pin. If it breaks, the loop is wrong, not the test. - **A learner's `write` assigns into `table.cumulative_regret` and never rebinds it.** The collector's workers read that array through a shared block; a rebound array leaves them walking stale regrets. @@ -76,9 +77,14 @@ itself is complete; what remains is the paid run and its grades. `Net`'s seven-output linear head and the freeze is the softmax at each key. `net.predict_by_point` is the one whole-game read `materialise`, Π's freeze and the distillation freeze share. -- A memory row is 24 bytes (`row: int32`, `t: int32`, `target: float32 × 4`); - the run's reservoirs are 50 million rows, 1.2 GB each. A capacity change is - a different run, not a resume. +- A memory row is 28 bytes (`row: int32`, `t: int32`, `target: float32 × 4`, + `weight: float32`); the run's reservoirs are 50 million rows, 1.4 GB each. A + capacity change is a different run, not a resume. The weight is how many + sampled rows the row stands for: 1 for every row a walk offers (what `merge` + stamps when given none), and the targeted-sampling plan's exact rows carry + `(1 − α) · K · arrival`. The refit and the oracle weigh a row by t × weight, + so a memory of weight-one rows is today's run bit for bit, and a memory file + saved before the column existed loads with ones. ## Running it diff --git a/src/drawmaha_solver/deepcfr/__init__.py b/src/drawmaha_solver/deepcfr/__init__.py index 31481960..73399642 100644 --- a/src/drawmaha_solver/deepcfr/__init__.py +++ b/src/drawmaha_solver/deepcfr/__init__.py @@ -13,9 +13,10 @@ - `net`: the MLP, chunked prediction into NumPy, and the save file. - `materialise`: two seats' nets into `cumulative_regret`, Deep CFR's eq. (3) obtained by what is written rather than by changing the ledger. -- `fit`: the from-scratch refit on (row, t, target) arrays. +- `fit`: the from-scratch refit on (row, t, target, weight) arrays. - `reservoir`: the memories those arrays come from — Algorithm R over - (row, t, target) rows, a counter-based stream per phase, an exact resume. + (row, t, target, weight) rows, a counter-based stream per phase, an exact + resume. - `collect`: K lockstep walks for one seat on W spawned processes, returned as the rows a phase offers the memories. - `train`: the loop that calls them in order, iteration after iteration — diff --git a/src/drawmaha_solver/deepcfr/fit.py b/src/drawmaha_solver/deepcfr/fit.py index d954bc48..5624db8c 100644 --- a/src/drawmaha_solver/deepcfr/fit.py +++ b/src/drawmaha_solver/deepcfr/fit.py @@ -1,26 +1,35 @@ -"""The from-scratch refit: a fresh net fitted to (row, t, target) rows. +"""The from-scratch refit: a fresh net fitted to (row, t, target, weight) rows. Deep CFR does not update a net between iterations; it throws the net away -and fits a new one to everything the memory holds. The memory is three +and fits a new one to everything the memory holds. The memory is four arrays the same length: `rows`, the table row each sample was taken at (the net never sees the row number — `Layout.features` turns it into the 136 floats the player could see); `t`, the iteration the sample was taken -on; and `targets`, what the net should say at that row's legal columns, -`MAX_WIDTH` wide with padding past the row's width. Nothing here knows -what a target IS — a sampled regret, an action probability — so the same -fit serves the regret nets and the policy net. +on; `targets`, what the net should say at that row's legal columns, +`MAX_WIDTH` wide with padding past the row's width; and `weight`, how many +sampled rows the row stands for, 1 for a walk's own sample (see +`reservoir.py` for the rows that are not one). Nothing here knows what a +target IS — a sampled regret, an action probability — so the same fit +serves the regret nets and the policy net. **The loss.** Per row, the squared error summed over its legal columns, read off the head through `features.legal_gather()`; the padding is masked out BEFORE the subtraction, so a padded column holding NaN is never looked -at and never leaks into a gradient. Each row is weighted by its iteration -t, divided by the batch's mean t so the loss keeps one scale as the run +at and never leaks into a gradient. Each row is weighted by t × weight — +its iteration times how many samples it stands for — divided by the +batch's mean of t × weight so the loss keeps one scale as the run lengthens, and the batch loss is the mean over rows: a 2-wide and a 4-wide -spot each count as one row. The weight is Deep CFR's (Brown, Lerer, Gross & +spot each count as one row. The t is Deep CFR's (Brown, Lerer, Gross & Sandholm 2019), borrowed from Linear CFR (Brown & Sandholm 2019): the early iterations' strategies are far from equilibrium, so the regrets sampled against them say the least about the game, and a weight that -grows with t lets the later samples outvote them. +grows with t lets the later samples outvote them. The weight multiplies t +rather than the target, so it changes how much a row counts and never +what the row says; at weight 1 throughout it is t / mean(t) bit for bit +(t · 1.0 is t in float32), and a row of weight 0 contributes nothing. A +batch whose every row weighs 0 has a mean of 0 to divide by; its loss is +0 rather than 0/0, so a tiny memory under the boost cannot end a fit as +"diverged" on one unlucky draw. **The optimiser.** Adam at the spec's learning rate, the gradient's norm clipped to the spec's bound before each step so one wild sample cannot @@ -49,7 +58,7 @@ feature halves and the legal gather go to the device once per fit and each batch's features are gathered there, so no step goes back through numpy. That copy holds the whole memory a second time for the fit's duration -(rows, t, and `MAX_WIDTH` float32 targets per row), so peak memory is +(rows, t, weight, and `MAX_WIDTH` float32 targets per row), so peak memory is about twice the memory's; unified memory makes the copy cheap, not free. Only the order of the arithmetic changes, so it is the same rule but not the same net. Adam divides by √v, so a near-zero gradient component turns @@ -116,34 +125,51 @@ def __post_init__(self) -> None: # The loss # --------------------------------------------------------------------------- -def batch_loss(net: Net, rows: np.ndarray, t: np.ndarray, targets: np.ndarray) -> torch.Tensor: +def batch_loss( + net: Net, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray | None = None, +) -> torch.Tensor: """One batch's loss, a scalar tensor: what every step minimises. Public so a test and PR 4's logging read the same number the optimiser - does. Builds the batch's features on the fly from `layout()`. Assumes - what `fit_from_scratch` checks: every t is >= 1, so the batch's mean t - the weights are divided by is positive. + does. Builds the batch's features on the fly from `layout()`. `weight` + left out is a batch of plain samples, every row weighing 1: a default + here, where a caller hands arrays it built itself, while `Learner.fit` + in `train.py` has none, because that is where a memory's column could + be dropped by accident. Assumes what `fit_from_scratch` checks: every t + is >= 1, every weight finite and non-negative, and t × weight within + float32's range. """ built = layout() point, _ = built.locate_rows(rows) gather = torch.from_numpy(legal_gather()[point].astype(np.int64)) + if weight is None: + weight = np.ones(len(rows), dtype=np.float32) return _weighted_loss( net(torch.from_numpy(built.features(rows))), gather, torch.from_numpy(np.asarray(targets, dtype=np.float32)), torch.from_numpy(np.asarray(t, dtype=np.float32)), + torch.from_numpy(np.asarray(weight, dtype=np.float32)), ) def _weighted_loss( - outputs: torch.Tensor, gather: torch.Tensor, targets: torch.Tensor, t: torch.Tensor + outputs: torch.Tensor, + gather: torch.Tensor, + targets: torch.Tensor, + t: torch.Tensor, + weight: torch.Tensor, ) -> torch.Tensor: """The loss from the net's outputs on: the one copy both the cpu and the device fit call. `outputs` (b, head) the net's raw head, `gather` (b, MAX_WIDTH) each - row's ledger columns with `NO_ACTION` padding, `targets` (b, MAX_WIDTH) - and `t` (b,) float32, all on one device. Everything before this — the - features, the gather's rows — differs between the paths only in where - it is looked up, never in what it is. + row's ledger columns with `NO_ACTION` padding, `targets` (b, MAX_WIDTH), + `t` (b,) and `weight` (b,) float32, all on one device. Everything + before this — the features, the gather's rows — differs between the + paths only in where it is looked up, never in what it is. """ legal = gather != NO_ACTION predicted = outputs.gather(1, gather.clamp(min=0)) @@ -151,7 +177,11 @@ def _weighted_loss( # survive `where` in the backward pass, where 0 * NaN is NaN. target = torch.where(legal, targets, 0.0) error = torch.where(legal, (predicted - target) ** 2, 0.0).sum(dim=1) - weight = t / t.mean() + scale = t * weight + # A batch of weight-0 rows alone has a mean of 0; clamped, its loss is 0 + # and not NaN. Any positive mean is far above `tiny`, so the clamp + # returns it unchanged and the division is the plain one. + weight = scale / scale.mean().clamp(min=torch.finfo(scale.dtype).tiny) return (weight * error).mean() # --------------------------------------------------------------------------- @@ -162,6 +192,7 @@ def fit_from_scratch( rows: np.ndarray, t: np.ndarray, targets: np.ndarray, + weight: np.ndarray | None = None, *, seed: tuple[int, ...], config: FitConfig = FitConfig(), @@ -171,13 +202,17 @@ def fit_from_scratch( `rows` (n,) int table rows, `t` (n,) iteration numbers >= 1 (an integer-valued float is accepted: t enters only as a float32 weight), `targets` (n, MAX_WIDTH) float in ledger-column order with padding - ignored. + ignored, `weight` (n,) finite and >= 0, or None for a memory of plain + samples, every row weighing 1. Refused by name before anything is built: - - a `rows` or `t` that is not 1-D, or non-integer rows; + - a `rows`, `t` or `weight` that is not 1-D, or non-integer rows; - mismatched lengths, or an empty memory; - a non-finite t, or one below 1; + - a non-finite or negative weight, one whose t × weight overflows + float32 (the loss's scale), or weights with no positive entry + (nothing to fit, as an empty memory is); - targets not `MAX_WIDTH` wide, or non-finite at a legal column; - a row outside the table. @@ -191,11 +226,16 @@ def fit_from_scratch( # refused by name, not left to broadcasting. if rows.ndim != 1 or t.ndim != 1: raise ValueError(f"rows and t must be 1-D, got shapes {rows.shape} and {t.shape}") + with np.errstate(over="ignore"): # a weight past float32's range is refused below as the inf it became + weight = np.ones(len(rows), dtype=np.float32) if weight is None else np.asarray(weight, dtype=np.float32) + if weight.ndim != 1: + raise ValueError(f"weight must be 1-D, got shape {weight.shape}") if not np.issubdtype(rows.dtype, np.integer): raise ValueError(f"rows must be integer row numbers, got dtype {rows.dtype}") - if not len(rows) == len(t) == len(targets): + if not len(rows) == len(t) == len(targets) == len(weight): raise ValueError( - f"rows, t and targets must have the same length, got {len(rows)}, {len(t)}, {len(targets)}" + "rows, t, targets and weight must have the same length, " + f"got {len(rows)}, {len(t)}, {len(targets)}, {len(weight)}" ) if len(rows) == 0: raise ValueError("the memory is empty: nothing to fit") @@ -205,6 +245,20 @@ def fit_from_scratch( raise ValueError("every t must be finite") if t.min() < 1: raise ValueError(f"every t must be an iteration number >= 1, got a minimum of {t.min()}") + # Checked after the cast to float32, so a float64 weight past float32's + # range is caught as the inf it became, not passed as the finite number + # it was given as. + if not np.isfinite(weight).all() or weight.min() < 0: + raise ValueError("every weight must be finite and non-negative in float32") + # The loss scales a row by t × weight in float32; a product past its + # range would reach the batch as inf and the loss as NaN. + with np.errstate(over="ignore"): + if not np.isfinite(t.astype(np.float32) * weight).all(): + raise ValueError("t × weight must fit float32") + # Every row inert: the batches would all be the weightless case the + # loss guards, and 4,000 steps of zero gradient would "fit" nothing. + if not (weight > 0).any(): + raise ValueError("no row carries weight: nothing to fit") rows = rows.astype(np.int64) point, _ = layout().locate_rows(rows) # raises IndexError for a row outside the table # Only the legal columns are checked: the padding may hold anything. @@ -215,7 +269,7 @@ def fit_from_scratch( # config can never quietly run the cpu loop; the device loop checks only # mps's availability, so a new name must add its own check there. if config.device != "cpu": - return _fit_on_device(rows, t, targets, folded, config, device=config.device) + return _fit_on_device(rows, t, targets, weight, folded, config, device=config.device) threads_before = torch.get_num_threads() torch.set_num_threads(config.threads) try: @@ -229,7 +283,7 @@ def fit_from_scratch( # (the cpu-parity test in test_fit.py is the guard). picked = torch.randint(len(rows), (config.batch,), generator=draw).numpy() optimiser.zero_grad() - batch_loss(net, rows[picked], t[picked], targets[picked]).backward() + batch_loss(net, rows[picked], t[picked], targets[picked], weight[picked]).backward() clip_grad_norm_(net.parameters(), config.clip) optimiser.step() finally: @@ -245,6 +299,7 @@ def _fit_on_device( rows: np.ndarray, t: np.ndarray, targets: np.ndarray, + weight: np.ndarray, folded: int, config: FitConfig, *, @@ -274,6 +329,7 @@ def _fit_on_device( gather_all = torch.from_numpy(legal_gather().astype(np.int64)).to(device) memory_rows = torch.from_numpy(rows).to(device) memory_t = torch.from_numpy(np.asarray(t, dtype=np.float32)).to(device) + memory_weight = torch.from_numpy(np.asarray(weight, dtype=np.float32)).to(device) # `from_numpy` warns on a read-only array (a memory loaded from a file # can be one); `require` copies only in that case. memory_targets = torch.from_numpy(np.require(targets, requirements="W")).to(device) @@ -292,7 +348,11 @@ def _fit_on_device( position = batch_rows - row_offset[point] features = torch.cat([private[private_offset[point] + position], public[point]], dim=1) loss = _weighted_loss( - net(features), gather_all[point], memory_targets[picked], memory_t[picked] + net(features), + gather_all[point], + memory_targets[picked], + memory_t[picked], + memory_weight[picked], ) optimiser.zero_grad() loss.backward() diff --git a/src/drawmaha_solver/deepcfr/policy.py b/src/drawmaha_solver/deepcfr/policy.py index 85dd6e7a..cd9e045f 100644 --- a/src/drawmaha_solver/deepcfr/policy.py +++ b/src/drawmaha_solver/deepcfr/policy.py @@ -23,7 +23,9 @@ **The fit.** `fit_from_scratch` on the memory's `contents()`; the targets are σ rows zero-padded to `TARGET_WIDTH`, and `batch_loss` masks the -padding before the subtraction, as it does for advantage rows. Nothing in +padding before the subtraction, as it does for advantage rows. The policy +memory's weights are all 1 — the opponent's walks are the only thing that +offers σ rows — so the column is carried and changes nothing here. Nothing in the fit knows the target is a probability, so a regret memory handed by mistake would fit in silence; `fit_policy` refuses one by checking that every row is a distribution at its legal columns. A fresh net every time, @@ -85,13 +87,16 @@ def fit_policy( rows: np.ndarray, t: np.ndarray, targets: np.ndarray, + weight: np.ndarray | None = None, *, seed: int, iteration: int, config: FitConfig | None = None, ) -> Net: - """A fresh net fit to the policy memory's `(rows, t, targets)` at `iteration`, under `policy_seed`. + """A fresh net fit to the policy memory's `(rows, t, targets, weight)` at `iteration`, under `policy_seed`. + `weight` is the memory's fourth column, positional so the memory's + `contents()` unpacks straight in; None is a memory of plain samples. `config` defaults to `FitConfig()`, the spec's refit. Refuses, before anything is built, a targets row that is not a distribution at its legal columns — a regret memory handed in place of the policy memory. @@ -101,7 +106,7 @@ def fit_policy( targets = np.asarray(targets, dtype=np.float32) _require_distributions(np.asarray(rows), targets) return fit_from_scratch( - rows, t, targets, seed=policy_seed(seed, iteration), config=config or FitConfig() + rows, t, targets, weight, seed=policy_seed(seed, iteration), config=config or FitConfig() ) def _require_distributions(rows: np.ndarray, targets: np.ndarray) -> None: diff --git a/src/drawmaha_solver/deepcfr/reservoir.py b/src/drawmaha_solver/deepcfr/reservoir.py index 49cf834b..a51149c7 100644 --- a/src/drawmaha_solver/deepcfr/reservoir.py +++ b/src/drawmaha_solver/deepcfr/reservoir.py @@ -1,13 +1,27 @@ -"""The memories: Algorithm R over (row, t, target) rows, with an exact resume. +"""The memories: Algorithm R over (row, t, target, weight) rows, with an exact resume. Deep CFR trains each iteration's net on a MEMORY of what the walks saw: for the advantage memory of seat p, every traverser visit's regret sample r = u − ⟨σ, u⟩ at some table row; for the policy memory, every opponent visit's σ. A row of -either memory is 24 bytes — `row: int32` (which table row), `t: int32` (which -iteration offered it, the refit's weight) and `target: float32 × 4` (r or σ at -the row's legal columns, zero-padded to the widest ledger, 4). Over 2,000 -iterations the walks offer about 280 million advantage rows per seat; the -memory keeps a capacity C of them, 50 million in the run. +either memory is 28 bytes — `row: int32` (which table row), `t: int32` (which +iteration offered it), `target: float32 × 4` (r or σ at the row's legal +columns, zero-padded to the widest ledger, 4) and `weight: float32` (how many +sampled rows this one stands for). The refit weighs a row by t × weight. Over +2,000 iterations the walks offer about 280 million advantage rows per seat; +the memory keeps a capacity C of them, 50 million in the run. + +**The weight.** Every row the walk offers is one sample and weighs 1, which is +what `merge` stamps when no weight is given, so a memory of walk rows alone is +what it was before the column existed. The column is for rows that are NOT one +sample: the targeted-sampling plan's boost stage adds, at a chosen infoset, one +row holding the EXACT regret under the current strategy, weighted +(1 − α) · K · arrival, the number of sampled rows the walk would have put there +in expectation, so the infoset's expected total weight per iteration is +unchanged; and it turns the walk's own rows there down to α. A weight of 0 is a +row the refit must not learn from, kept in the memory so the reservoir sees the +same offers as a run without the boost. The weight rides the same slot as its +row through eviction, save and load, and a file written before the column +existed loads with ones, so a saved run resumes as the run it was. **Which C of the n rows ever offered?** A uniform sample without replacement — every offer resident with probability C/n — so that the refit sees the @@ -34,10 +48,11 @@ fifth word, the ASCII constant "RESV", keeps a reservoir stream from ever equalling a walk's four-word stream. -The three arrays are `np.zeros` at construction, which the kernel pages in -lazily: a 50M-row memory costs 1.2 GB only as it fills, nothing when three of +The four arrays are `np.zeros` at construction, which the kernel pages in +lazily: a 50M-row memory costs 1.4 GB only as it fills, nothing when three of them are built on a laptop. `contents()` is the refit's input — read-only views -of the resident rows, the shape `fit_from_scratch(rows, t, targets, …)` takes. +of the resident rows, the shape `fit_from_scratch(rows, t, targets, weight, …)` +takes. """ from __future__ import annotations @@ -57,9 +72,13 @@ # has four words, so no reservoir stream can equal one. _STREAM_TAG = 0x52455356 -# What a reservoir file holds. `offered` marks a file as a reservoir: no other -# checkpoint in the repo records a count of offers. +# What every reservoir file holds. `offered` marks a file as a reservoir: no +# other checkpoint in the repo records a count of offers. _FILE_KEYS = ("rows", "t", "targets", "offered", "capacity") +# Written by every `save` since the column existed, and read as ones when +# absent: a file without it predates the column, and every row it holds was +# one sample. +_WEIGHT_KEY = "weight" class Memory(IntEnum): """Which memory a stream feeds; the word that tells the two apart in the seed.""" @@ -92,6 +111,7 @@ def __init__(self, capacity: int) -> None: self._rows = np.zeros(self._capacity, dtype=np.int32) self._t = np.zeros(self._capacity, dtype=np.int32) self._targets = np.zeros((self._capacity, TARGET_WIDTH), dtype=np.float32) + self._weight = np.zeros(self._capacity, dtype=np.float32) @property def capacity(self) -> int: @@ -108,14 +128,25 @@ def size(self) -> int: """How many rows are resident: min(n, capacity).""" return min(self._offered, self._capacity) - def merge(self, rows: np.ndarray, targets: np.ndarray, *, t: int, rng) -> None: + def merge( + self, + rows: np.ndarray, + targets: np.ndarray, + *, + t: int, + rng, + weight: np.ndarray | None = None, + ) -> None: """Offer one phase's rows, all stamped `t`, drawing from `rng`. - `rng` needs `Generator.integers(low, high)` with an array `high`: one - draw per late offer. Rows below capacity are kept outright and draw - nothing, so a memory still filling consumes no randomness. + `weight` is one finite, non-negative float per row, with t × weight + within float32's range; left out, every row weighs 1, the walk's own + sample. `rng` needs + `Generator.integers(low, high)` with an array `high`: one draw per + late offer. Rows below capacity are kept outright and draw nothing, + so a memory still filling consumes no randomness. """ - rows, targets = _validate_offer(rows, targets, t) + rows, targets, weight = _validate_offer(rows, targets, t, weight) count = rows.shape[0] capacity = self._capacity # Every offer's global index among all offers ever made. @@ -136,16 +167,17 @@ def merge(self, rows: np.ndarray, targets: np.ndarray, *, t: int, rng) -> None: self._rows[slot[winners]] = rows[winners] self._t[slot[winners]] = t self._targets[slot[winners]] = targets[winners] + self._weight[slot[winners]] = weight[winners] self._offered += count - def contents(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """The resident rows as read-only views: `(rows, t, targets)`, the refit's input. + def contents(self) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: + """The resident rows as read-only views: `(rows, t, targets, weight)`, the refit's input. Live views, not a snapshot: a later `merge` shows through them. The loop takes them after the phase's merge and hands them to the refit. """ size = self.size - views = (self._rows[:size], self._t[:size], self._targets[:size]) + views = (self._rows[:size], self._t[:size], self._targets[:size], self._weight[:size]) for view in views: view.setflags(write=False) return views @@ -159,7 +191,7 @@ def save(self, path: Path) -> None: crash mid-write leaves the previous file whole. """ path = Path(path) - rows, t, targets = self.contents() + rows, t, targets, weight = self.contents() temporary = path.with_name(path.name + ".tmp") with temporary.open("wb") as file: np.savez( @@ -167,6 +199,7 @@ def save(self, path: Path) -> None: rows=rows, t=t, targets=targets, + weight=weight, offered=np.int64(self._offered), capacity=np.int64(self._capacity), ) @@ -179,7 +212,14 @@ def load(cls, path: Path) -> Reservoir: """A memory as `save` left it, ready for the next `merge`. Refuses a file that is not a reservoir (no `offered`) and one whose - resident rows do not match its counts, naming the file either way. + resident rows do not match its counts, naming the file either way. A + file with no `weight` is from before the column existed: its rows + were all plain samples and load weighing one. A `weight` the column + would not have stored — not float32, not one per row, negative, + non-finite, or overflowing t × weight in float32 — is refused naming + the file: `merge` never writes one, so it is a corrupt or hand-edited + file, and `fit_from_scratch` would otherwise be the first to say so, + without the file's name. """ path = Path(path) with np.load(path, allow_pickle=False) as saved: @@ -188,30 +228,42 @@ def load(cls, path: Path) -> Reservoir: raise ValueError(f"{path} is not a reservoir checkpoint (no {missing[0]!r})") capacity, offered = int(saved["capacity"]), int(saved["offered"]) rows, t, targets = saved["rows"], saved["t"], saved["targets"] + weight = saved[_WEIGHT_KEY] if _WEIGHT_KEY in saved.files else None # The constructor would refuse this too, but without the file's name. if capacity < 1: raise ValueError(f"{path}: capacity must be a positive int, got {capacity}") memory = cls(capacity) size = min(offered, capacity) - if ( - offered < 0 - or rows.shape != (size,) - or t.shape != (size,) - or targets.shape != (size, TARGET_WIDTH) - ): + if offered < 0 or rows.shape != (size,) or t.shape != (size,) or targets.shape != (size, TARGET_WIDTH): raise ValueError( - f"{path} holds {rows.shape[0]} rows, but {offered} offers into a capacity " + f"{path} holds {rows.shape} rows, but {offered} offers into a capacity " f"of {capacity} means {size} resident" ) + if weight is None: + weight = np.ones(size, dtype=np.float32) + elif weight.shape != (size,): + raise ValueError(f"{path} holds {size} rows but a weight of shape {weight.shape}") # The arrays are poured in as they are: a float `rows` would be # truncated into the int32 slots without a word. - expected = (("rows", rows, np.int32), ("t", t, np.int32), ("targets", targets, np.float32)) + expected = ( + ("rows", rows, np.int32), + ("t", t, np.int32), + ("targets", targets, np.float32), + ("weight", weight, np.float32), + ) for name, array, dtype in expected: if array.dtype != dtype: raise ValueError(f"{path}: {name} must be {dtype.__name__}, got {array.dtype}") + # What `merge` refuses on the way in, refused on the way back as well. + if not (np.isfinite(weight).all() and (weight >= 0).all()): + raise ValueError(f"{path}: every weight must be finite and non-negative") + with np.errstate(over="ignore"): + if not np.isfinite(t.astype(np.float32) * weight).all(): + raise ValueError(f"{path}: t × weight must fit float32") memory._rows[:size] = rows memory._t[:size] = t memory._targets[:size] = targets + memory._weight[:size] = weight memory._offered = offered return memory @@ -223,14 +275,17 @@ def _is_int(value: object) -> bool: # A bool is an int to Python and never what a caller meant. return isinstance(value, (int, np.integer)) and not isinstance(value, bool) -def _validate_offer(rows: np.ndarray, targets: np.ndarray, t: int) -> tuple[np.ndarray, np.ndarray]: - """The phase's rows as int32 and its targets as float32, or a ValueError naming the fault.""" +def _validate_offer( + rows: np.ndarray, targets: np.ndarray, t: int, weight: np.ndarray | None +) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """The phase's rows as int32, its targets and weights as float32, or a ValueError naming the fault.""" rows = np.asarray(rows) targets = np.asarray(targets) # A row is a table row number: a float or a negative is a caller handing # the wrong array. if rows.ndim != 1 or not np.issubdtype(rows.dtype, np.integer): raise ValueError(f"rows must be a 1-D integer array, got {rows.dtype} {rows.shape}") + weight = np.ones(rows.shape[0], dtype=np.float32) if weight is None else np.asarray(weight) if rows.size and rows.min() < 0: raise ValueError(f"rows must not be negative, got {int(rows.min())}") # Rows are stored as int32; a bigger one would wrap silently in `astype`. @@ -245,6 +300,12 @@ def _validate_offer(rows: np.ndarray, targets: np.ndarray, t: int) -> tuple[np.n "rows and targets must have the same length, " f"got {rows.shape[0]} and {targets.shape[0]}" ) + if weight.shape != (rows.shape[0],): + raise ValueError(f"weight must be shaped ({rows.shape[0]},), one per row, got {weight.shape}") + # A weight is a count of samples: a bool, a complex or an object array is + # a caller handing the wrong array, not a count. + if not (np.issubdtype(weight.dtype, np.floating) or np.issubdtype(weight.dtype, np.integer)): + raise ValueError(f"weight must be a float or int array, got dtype {weight.dtype}") # Iterations start at 1; the refit weights a row by its t, so 0 would # erase it and a negative would pull against the others. if not _is_int(t) or t < 1: @@ -253,4 +314,16 @@ def _validate_offer(rows: np.ndarray, targets: np.ndarray, t: int) -> tuple[np.n # below the bound, which is here so the stamp can never wrap in silence. if t >= 2**31: raise ValueError(f"t must fit int32, got {t!r}") - return rows.astype(np.int32, copy=False), targets.astype(np.float32, copy=False) + # The column is float32, so the weight is checked AFTER the cast: a + # float64 weight past float32's range is finite as given and inf once + # stored. The refit multiplies a row's error by t × weight in float32: a + # negative weight would pull the net AWAY from the row's target, a NaN + # or an inf (the product's, if not the weight's own) would spread through + # the batch's mean into every row's gradient. + with np.errstate(over="ignore"): # the overflow is what is being looked for + weight = weight.astype(np.float32, copy=False) + if weight.size and not (np.isfinite(weight).all() and weight.min() >= 0): + raise ValueError("every weight must be finite and non-negative in float32") + if weight.size and not np.isfinite(np.float32(t) * weight).all(): + raise ValueError(f"t × weight must fit float32: at t = {t} a weight of {weight.max()} overflows") + return rows.astype(np.int32, copy=False), targets.astype(np.float32, copy=False), weight diff --git a/src/drawmaha_solver/deepcfr/train.py b/src/drawmaha_solver/deepcfr/train.py index 9aba3170..d6f0ddac 100644 --- a/src/drawmaha_solver/deepcfr/train.py +++ b/src/drawmaha_solver/deepcfr/train.py @@ -12,7 +12,7 @@ rows = collector.collect(seat=p, t=t, walks=K) advantage[p].merge(rows.advantage_rows, rows.advantage_targets, t=t, rng=stream(ADVANTAGE)) policy.merge(rows.policy_rows, rows.policy_targets, t=t, rng=stream(POLICY)) - nets[p] = learner.fit(*advantage[p].contents(), seed=(seed, t, p)).model + nets[p] = learner.fit(*advantage[p].contents(), seed=(seed, t, p)).model # rows, t, targets, weight — plus everything a 15-hour run needs around it: the resume state, grid-mark saves, a progress line per iteration, and a graceful stop on a signal or a @@ -33,8 +33,11 @@ imports `fit`, `materialise` and `net` inside those methods. That is what lets the whole-loop test run in the default suite with no torch: the `OracleLearner` below is a "perfect net" that returns, at every row, the -t′-weighted sum of every regret sample the memory holds there, and with it -Deep CFR collapses into rung 3's tabular LCFR. The one thing the loop asks +sum over every regret sample the memory holds there of t′ × weight × r, and +with it Deep CFR collapses into rung 3's tabular LCFR. The weight is the +memory's fourth column (`reservoir.py`): 1 for every row a walk offers, so +until the targeted-sampling plan's boost stage injects rows that stand for +more or fewer samples than one, it changes nothing. The one thing the loop asks of a `write` is that it assigns INTO `table.cumulative_regret` and never rebinds it: the collector's workers read that array through a shared block, and a rebound array would leave them walking stale regrets. @@ -53,7 +56,10 @@ reordered row) still breaks that — and to float32 precision against the untouched `train_lockstep`. The oracle repeats LCFR's float operations in LCFR's order (per t′ in increasing order, `R[cols] += t′ · totals`) because -a single Σ t′·r over all rows would round differently from t = 2 on. +a single Σ t′·r over all rows would round differently from t = 2 on. The +weight multiplies each sample before that sum, in float64, where × 1.0 is +exact: a memory of weight-one rows banks the same bits as before the column +existed, which is what keeps the pin a pin. **The resume state** is three memories, the two latest nets, the file of every observer that carries state (PR 5's `ExactAverage`: an observer with @@ -311,14 +317,23 @@ class Fitted: class Learner(Protocol): """The one seam between the loop and the model. - `fit` takes a memory's `contents()` and the fit's seed `(seed, t, seat)`; - `write` plays `models[seat]` for both seats by assigning into - `table.cumulative_regret`; `save` and `load` round-trip one model - through one file, for the resume state and the kept nets. + `fit` takes a memory's `contents()` — rows, t, targets and the weight + of each row, how many samples it stands for — and the fit's seed + `(seed, t, seat)`; `write` plays `models[seat]` for both seats by + assigning into `table.cumulative_regret`; `save` and `load` round-trip + one model through one file, for the resume state and the kept nets. + The weight is positional and has no default here, so no learner can + take the memory and drop its weights by accident. """ def fit( - self, rows: np.ndarray, t: np.ndarray, targets: np.ndarray, *, seed: tuple[int, int, int] + self, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray, + *, + seed: tuple[int, int, int], ) -> Fitted: ... def write(self, table: PackedTable, models: Sequence[object | None]) -> None: ... @@ -388,7 +403,10 @@ def step( `collector` is a `Collector` (or anything with its `table` and `collect`); the iteration is `run.t + 1`, and `run.t` advances once both phases are done. A raise mid-way leaves `run` half-advanced (seat 0's - net and memories updated, `t` not), which `run_deep` never saves. + net and memories updated, `t` not), which `run_deep` never saves. The + walks' rows are merged without a weight and so weigh one each; the + targeted-sampling plan's boost stage is what will offer rows that weigh + otherwise, after the natural rows so these keep their stream positions. """ t = run.t + 1 config = run.config @@ -1037,7 +1055,13 @@ def __init__(self, config: DeepConfig) -> None: self._fit_config = None def fit( - self, rows: np.ndarray, t: np.ndarray, targets: np.ndarray, *, seed: tuple[int, int, int] + self, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray, + *, + seed: tuple[int, int, int], ) -> Fitted: import torch @@ -1047,12 +1071,18 @@ def fit( if self._config.rule != "linear": targets = rule_targets(targets, t, now=seed[1], rule=self._config.rule) t = np.ones(len(t), dtype=np.float32) + # The weight is a sample weight, how much the row counts, and never + # touches the target: the pot scaling and the rules act on what the + # row says, the weight on how loudly, and the two compose. targets = pot_scaled(rows, targets, power=self._config.pot_power) - net = fit_from_scratch(rows, t, targets, seed=seed, config=config) + net = fit_from_scratch(rows, t, targets, weight, seed=seed, config=config) draw = torch.Generator().manual_seed(_fold_seed((*seed, _HELD_TAG))) held = torch.randint(len(rows), (config.batch,), generator=draw).numpy() + # The held loss is a number for the progress line (`StepStats.loss`) + # and nothing decides on it: a held batch of weight-0 rows alone + # reads 0, as the loss would, and says so there rather than NaN. with torch.inference_mode(): - loss = float(batch_loss(net, rows[held], t[held], targets[held]).item()) + loss = float(batch_loss(net, rows[held], t[held], targets[held], weight[held]).item()) return Fitted(net, loss) def write(self, table: PackedTable, models: Sequence[object | None]) -> None: @@ -1117,19 +1147,28 @@ class OracleModel: rows: np.ndarray # int32 (n,) t: np.ndarray # int32 (n,) targets: np.ndarray # float32 (n, TARGET_WIDTH) + weight: np.ndarray # float32 (n,) class OracleLearner: """The learner that makes the loop LCFR: its model is the memory, its write is LCFR's bank. - `fit` copies the memory. `write` puts at every row of each seat the - t′-weighted sum of the samples its model holds there, computed as LCFR - computes it — per t′ in increasing order, the samples of that t′ summed - per column in memory order with `np.add.at`, then `R[cols] += t′ · - totals` — so the bits match `bank_rows` through any number of - iterations, not only the first. The two models are summed one after - the other, which is only LCFR's order because a row belongs to one - acting seat and so appears in one model. A row no sample names stays - exactly 0.0. NumPy only, so gate 1 runs without torch. + `fit` copies the memory. `write` puts at every row of each seat the sum + of t′ × weight × sample over the samples its model holds there, computed + as LCFR computes it — per t′ in increasing order, the samples of that + t′ each multiplied by their weight in float64 and summed per column in + memory order with `np.add.at`, then `R[cols] += t′ · totals` — so the + bits match `bank_rows` through any number of iterations, not only the + first: × 1.0 is exact, so a memory of weight-one rows banks what it + banked before the column existed. A row of weight w banks w × r, so a + row of weight 0 banks nothing, and a row of weight 2 banks 2r, which is + what the same row offered twice banks when the two are the only samples + at the column in that t′ (0 + r + r is 2r exactly); with other samples + there, the two agree only up to rounding, since `np.add.at` sums the + column's samples one at a time and (a + r) + r is not a + 2r to the + bit. The two models are summed one after the other, which is + only LCFR's order because a row belongs to one acting seat and so + appears in one model. A row no sample names stays exactly 0.0. NumPy + only, so gate 1 runs without torch. """ def __init__(self) -> None: @@ -1137,9 +1176,17 @@ def __init__(self) -> None: self._starts: dict[int, tuple[np.ndarray, np.ndarray]] = {} def fit( - self, rows: np.ndarray, t: np.ndarray, targets: np.ndarray, *, seed: tuple[int, int, int] + self, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray, + *, + seed: tuple[int, int, int], ) -> Fitted: - return Fitted(OracleModel(np.array(rows), np.array(t), np.array(targets)), None) + return Fitted( + OracleModel(np.array(rows), np.array(t), np.array(targets), np.array(weight)), None + ) def write(self, table: PackedTable, models: Sequence[OracleModel | None]) -> None: starts, widths = self._layout(table) @@ -1153,7 +1200,11 @@ def write(self, table: PackedTable, models: Sequence[OracleModel | None]) -> Non width = widths[rows] columns = _columns(starts[rows], width) legal = np.arange(TARGET_WIDTH)[None, :] < width[:, None] - values = model.targets[at][legal].astype(np.float64) + # `legal` reads row by row, so each row's weight repeats once + # per legal column, in the same order as its values. + values = model.targets[at][legal].astype(np.float64) * np.repeat( + model.weight[at].astype(np.float64), width + ) touched, _, inverse = np.unique(columns, return_index=True, return_inverse=True) totals = np.zeros(touched.size) np.add.at(totals, inverse, values) @@ -1164,11 +1215,18 @@ def write(self, table: PackedTable, models: Sequence[OracleModel | None]) -> Non def save(self, model: OracleModel, path: Path) -> None: # Through an open file, so NumPy does not append `.npz` to the name. with Path(path).open("wb") as file: - np.savez(file, rows=model.rows, t=model.t, targets=model.targets) + np.savez(file, rows=model.rows, t=model.t, targets=model.targets, weight=model.weight) def load(self, path: Path) -> OracleModel: + # A model saved before the weight column existed held plain samples, + # one each, as a reservoir file without the key does. with np.load(Path(path), allow_pickle=False) as saved: - return OracleModel(saved["rows"], saved["t"], saved["targets"]) + weight = ( + saved["weight"] + if "weight" in saved.files + else np.ones(saved["rows"].shape[0], dtype=np.float32) + ) + return OracleModel(saved["rows"], saved["t"], saved["targets"], weight) def _layout(self, table: PackedTable) -> tuple[np.ndarray, np.ndarray]: if id(table) not in self._starts: diff --git a/src/drawmaha_solver/minidrawmaha/AGENTS.md b/src/drawmaha_solver/minidrawmaha/AGENTS.md index 56f5ded6..97aa5019 100644 --- a/src/drawmaha_solver/minidrawmaha/AGENTS.md +++ b/src/drawmaha_solver/minidrawmaha/AGENTS.md @@ -13,7 +13,14 @@ Three subpackages, and one module that stays alone on purpose: arrays), `infoset_table` (a ledger window over it), `mccfr` (external sampling, one hand per seat), `lockstep` (W hands per seat against one frozen σ; the reference), `parallel` (the same in W processes over shared - memory), `lockstep_run` (resume, grid checkpoints, progress). + memory), `lockstep_run` (resume, grid checkpoints, progress), and + `boost_target` (rung 4's boost target: extra samples of a high-noise row's + regret, each source below the row — the board card and both seats' + replacements k of n, the opponent's actions and range k draws — or + enumerated, as a `Sampling` names, unbiased, with `EXACT`, the sum over every opponent hand + and every physical entry of the key, as their limit; and the exact + probability a walk arrives there; rules-only, and it must never import + `exploitability.py`, which grades what it feeds). - `answer/` — everything built on the solved game. `strategy` (the frozen LCFR average, a release asset pinned by SHA-256), `analysis` (`minidraw-analysis`, writes `figures/rung3/`), `play` (`minidraw-play`), diff --git a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py new file mode 100644 index 00000000..8830f431 --- /dev/null +++ b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py @@ -0,0 +1,1112 @@ +"""The boost target: extra samples of a high-noise row's regret, unbiased, with the exact sum as their limit. + +Rung 4 learns every infoset from the regret samples the walks bank there, +`r = u − ⟨σ, u⟩` with one opponent hand and one line of opponent play drawn +per visit (`mccfr.traverse`, `lockstep.walk`). At a rare round-2 row a visit +comes once in thousands of iterations and its sample swings by chips, almost +all of it the opponent's hidden hand. This module draws more samples of that +same quantity at a chosen row: it starts at the row, faces the opponent's +whole range (or k hands of it), and plays +the hand out with each source of randomness below — the board card, both +seats' replacements, the opponent's actions — sampled k of n or enumerated, +as a `Sampling` names. With every source enumerated (`EXACT`) the target is +the exact conditional mean the walk's samples estimate, and that is what the +tests pin the sampled forms to. A later stage (the "boost") banks the target +at the weight the walk's samples would have carried there, so a targeted +infoset is learned from a few good samples instead of a handful of noisy +ones. Against the whole range the opponent's hidden hand — nearly all of a +walk sample's spread — is gone, but a low-k form is not lower-noise than a +walk sample by construction: a k-of-n fan here is public, so it can keep a +branch that is dead for the hand at hand (the board card among a throw's 12 +replacements, a held card among the 14 second-board cards) and scales the +live ones up to pay for it, where the walk deals only live cards. Unbiased +either way; how much noise a form leaves is measured, not assumed. + +**What is sampled, and why each stays unbiased.** Every sampled source +replaces a sum over n branches by k of them, so that the expectation over +the draw is the sum: + +- a chance fan (the second board card, 14 branches; a throw's replacement, + 12 per hand, for either seat) keeps k of its n branches without + replacement, each scaled n/k: each branch is kept with probability k/n; +- the opponent's actions send each opponent state's weight down k actions + drawn from its σ at 1/k each, instead of splitting it by σ: the expected + share of an action is σ(a); +- the opponent range draws k states per physical entry in proportion to + their weight, among those sharing no card with the entry's own state, each + carrying 1/k of the entry's total; +- `playouts` independent play-outs are averaged. + +The traverser's own actions are never sampled, as the walk never samples +them: the regret is defined against each of them. The arrival — the ratio's +denominator below — is always exact. It is the cheap half (the opponent's +reach, back-solved upward, never sampled), and an exact denominator under an +unbiased numerator keeps the ratio unbiased; a sampled one would not, since +the mean of a ratio is not the ratio of the means. Each row draws from +`default_rng([seed, t, seat, row, _STREAM_TAG])`, its own stream, independent +of the walk's, so a row's target depends on (seed, t, seat, row) and never on +which rows share the call. + +**What is computed.** For a row standing for the key I of seat `seat`: + + regret(I, a) = Σ_{b,h,o} w(b,h,o) · (v(a | b,h,o) − v(I | b,h,o)) / Σ_{b,h,o} w(b,h,o) + arrival(I) = Σ_{b,h,o} w(b,h,o) + w(b,h,o) = deal(h) · r_b[o] · compat_{b,h}[o] + +where (b, h) runs over every PHYSICAL entry of the key — a board row in the +order it came, and the seat's physical (hole, thrown card) — and o over the +opponent's physical states. `r_b[o]` is the reach of the deck and the +opponent at the row's public point on board b: the opponent's deal weight, +the chance probability of every card dealt along the path, and the +opponent's σ at each of its decisions, pushed through its draw. `compat` +is one where the two seats' cards are disjoint. `deal(h)` is the +traverser's own deal probability — seat 0 carries 1/100,100 per allowed +pair and seat 1 carries 1 (the grader's convention in `_weights`, restated +here) — and at a post-draw state it is the sum over the three pre-draw holes +that could have become it, because the key forgets which card was drawn and +the walk arrives through any of them. `v(a | b,h,o)` is the traverser's +expected chips below action a with both seats at σ and the deck averaged. + +**Why the ratio, and why the traverser's own σ never enters the weight.** +The walk enumerates the traverser's own actions, so how often it arrives at +I is the deck's and the opponent's doing alone: `arrival` is exactly the +expected number of visits per walk, and `regret` is the conditional mean of +the banked sample given a visit — `E[r | reach]`. Normalising by Σ w is what +makes the exact row commensurable with the sampled ones: a sampled row's +target is one draw of r, so its expectation is this ratio, and the boost's +row weight, `K · arrival · t`, restores the total weight the K walks would +have put there. Dropping the denominator, or weighting every target 1, +would be a "teleport" that over-weights the rare rows it lands on, which is +the bias the targeted-sampling plan (kept, like the rung-3 plan, outside +the repo on Alec's machine under `~/Desktop/Claude/poker/`) exists to +avoid. The traverser's own σ enters only inside `v(I) = ⟨σ, v(·)⟩`, the +regret's baseline, as it does in the walk. One caveat on "visits": a walk +can bank a row twice (about one traversal in a thousand, through the suit +relabelling), so `arrival` is the expected number of visits per walk, a +probability but for those, and `regret` is the mean banked sample per +visit. + +**Why every physical entry is summed.** A key is a suit orbit of one +picture — up to six physical (board, state) pairs, fewer when the picture +has a suit symmetry. In the full game a global suit relabelling maps the +whole game onto itself and σ is read in key space, so the six entries +carry identical numbers and one representative times the orbit size would +do. The module sums them anyway: a restricted deal set (`build(holes=)`, +which PR 4's oracle test and this module's own tests use) is not closed +under relabelling, and there the entries face different opponent ranges +and the sum is load-bearing. The orbit shortcut is an optimisation left for +the day the full game's cost matters, with the sum as its oracle. + +**Card removal, applied once and in the same place.** The numerator's +weights and the denominator's are the same `compat` vector, computed by +`_compat` from the two seats' card sets. Below the row nothing further is +masked by hand: a replacement the deck could not deal — the board card, a +card the other seat holds — leads to a terminal where the two seats share a +card and pays zero through the same compat, and the chance node's public +1/|stub| is then exactly the probability of each live replacement, because +the stub's size is public (the walk relies on the same fact). The board-2 +deal masks the opponent's states the same way the grader's levels do. + +**The frame.** Every vector is indexed by a seat's physical state, 455 +holes in deck order × 16 slots (slot 0 for nothing thrown, 1 + c for card +c), 7,280 entries, with the board as a batch axis: 15 rows after the first +card, 210 ORDERED pairs after the second. Canonical keys enter at one place, +a map from (board row, state) to a position in `private_keys(shape)`, used +to gather σ rows; the same map inverted lists each key's physical entries. +That is the grader's frame, re-derived here from `rules/` alone. + +**Rules-only, on purpose.** This module imports `rules/` and the packed +table and nothing from `exploitability.py`, which grades what the boost +feeds; a tree, a key map or a showdown table shared between the two would +let one bug certify itself. The tests hold the two descriptions to each +other instead: the back-solved reach equals the grader's carried vector on +every board of every point to 1e-10, and the regret and arrival equal a +world-by-world walk through `MiniState` and the mean of thousands of real +walks. + +**Cost, measured on this Mac (2026-11-06, a table with every action +positive, so nothing below a row prunes).** The description is built once +per process, 5–7 s — the key maps, the draw maps, the public tree with +parent links, the showdown ranks — and holds no showdown matrix: shares and +compatibility are computed per terminal from the holes' ranks and card sets, +folded over the opponent's 455 holes. `sigma` over the whole table takes +0.12 s. Exactly, a round-2 row costs about 2 ms, a draw row 160–230 ms and a +round-1 row 0.5–0.75 s, because the value below the last two expands over +the traverser's own replacements (36 states per throw) and the second board +card (14 boards). Under the walk's own sampling against the whole range +(`Sampling()`) a draw row +costs about 25–32 ms and a round-1 row about 100–130 ms: the board card and +the hero's replacements are what sampling cuts. A round-2 row costs more +sampled than exact, about 3–4 ms: against a whole range the opponent's +actions are split by σ in one array operation, and drawing them per state +only adds a draw and noise — action sampling earns its keep when +`opponent_range` is a few hands, where most lines carry no weight and are +skipped. `tests/minidrawmaha/test_boost_target.py` prints the three costs on +every run. The opponent's reach is memoised per call at every node with one +board card out that lies on the path to a decision (66: the 29 level-1 +decision points and 37 chance nodes, 15 boards each) and at every round-2 +opening decision (20: one per round-1 line and pair of draw counts, 210 +boards each), 58 KB a (node, board) pair, so the memo is at most +66 × 15 + 20 × 210 = 5,190 vectors, 302 MB — a count the test file +re-derives from the tree; a call over 2,240 round-2 rows peaked at about +260 MB. +""" + +from __future__ import annotations + +import numbers +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from functools import cache +from itertools import combinations, permutations +from typing import Any + +import numpy as np + +from drawmaha_solver.minidrawmaha.rules.cards import DECK, Card +from drawmaha_solver.minidrawmaha.rules.enumeration import ( + PublicPoint, + private_keys, + public_decision_points, +) +from drawmaha_solver.minidrawmaha.rules.game import ( + BETTING_ACTIONS, + DRAW_ACTIONS, + Action, + MiniState, + NodeKind, + canonical_picture, + chip_state, + draw_order, + throw_count, +) +from drawmaha_solver.minidrawmaha.rules.hands import HOLE_CARDS, inner_score, outer_score +from drawmaha_solver.minidrawmaha.solver.packed_table import PackedTable + +Hole = tuple[Card, ...] +SEATS = (0, 1) + +# The widest ledger: the draw's four throws. Regret rows are padded to it. +TARGET_WIDTH = max(len(DRAW_ACTIONS), len(BETTING_ACTIONS)) + +# The tail of a boost stream's seed, ASCII "BOOST": 39 bits, which +# SeedSequence takes as two 32-bit words. For seed, t, seat and row below +# 2³² a row's stream is six words, so it cannot equal a walk's four-word +# stream or the reservoir's five-word "RESV" one, and its last two words +# tell it from either at any size. +_STREAM_TAG = int.from_bytes(b"BOOST", "big") + +# --------------------------------------------------------------------------- +# What the play-out below a boosted row samples +# --------------------------------------------------------------------------- + +_SOURCES = ("opponent_range", "board_cards", "opponent_draws", "hero_draws", "opponent_actions") + +# The fans a chance count is read against: the second board card can be any +# of the 14 cards the first left, and a throw's replacement any of the 12 +# cards not in the thrower's hole (dead ones pay zero below). +_BOARD_FAN = len(DECK) - 1 +_REPLACEMENT_FAN = len(DECK) - HOLE_CARDS + +@dataclass(frozen=True, slots=True) +class Sampling: + """How many of each random thing below a boosted row one play-out takes; `None` takes all of them. + + - `playouts` — independent play-outs from the row, averaged. + - `opponent_range` — `None` plays against the whole range; k draws k + opponent hands per physical entry, with replacement, in proportion to + their weight, each carrying 1/k of the entry's total. + - `board_cards` — k of the 14 cards the second board card can be, + without replacement, each scaled 14/k. + - `opponent_draws`, `hero_draws` — k of the 12 replacements a throw can + bring, per hand, without replacement, each scaled 12/k. + - `opponent_actions` — `None` splits every opponent hand's weight over its + actions by σ; k sends it down k actions drawn from σ, 1/k each. + + A chance count at or above its fan is that source enumerated. The + defaults are the walk's own sampling below the row against the whole + opponent range; `EXACT` enumerates everything and is the oracle. The + traverser's own actions are never sampled — the walk enumerates them, and + the regret is defined against them — only the cards its throws bring. + """ + + playouts: int = 1 + opponent_range: int | None = None + board_cards: int | None = 1 + opponent_draws: int | None = 1 + hero_draws: int | None = 1 + opponent_actions: int | None = 1 + + def __post_init__(self) -> None: + for name in ("playouts", *_SOURCES): + value = getattr(self, name) + if value is None and name != "playouts": + continue + if isinstance(value, bool) or not isinstance(value, numbers.Integral) or value < 1: + allowed = "a positive int" if name == "playouts" else "a positive int, or None for all" + raise ValueError(f"Sampling.{name} must be {allowed}, got {value!r}") + if self.is_exact and self.playouts != 1: + raise ValueError( + f"Sampling draws nothing (every count None, or a chance count at or above its fan), so every play-out is the same; playouts must be 1, got {self.playouts}" + ) + + @property + def is_exact(self) -> bool: + """Whether nothing is drawn: every count `None`, or a chance count at or above its fan.""" + return ( + self.opponent_range is None + and self.opponent_actions is None + and all( + k is None or k >= fan + for k, fan in ( + (self.board_cards, _BOARD_FAN), + (self.opponent_draws, _REPLACEMENT_FAN), + (self.hero_draws, _REPLACEMENT_FAN), + ) + ) + ) + +EXACT = Sampling( + playouts=1, opponent_range=None, board_cards=None, opponent_draws=None, hero_draws=None, opponent_actions=None +) + +@dataclass(frozen=True, slots=True, eq=False) +class _Draws: + """The play-out's one source of randomness: under `EXACT` every method hands back everything and the RNG is never touched.""" + + sampling: Sampling + rng: np.random.Generator | None + + def subset(self, k: int | None, n: int, shape: tuple[int, ...]) -> np.ndarray | None: + """k of n positions for every index of `shape`, without replacement; `None` for all n.""" + if k is None or k >= n: + return None + return np.argsort(self.rng.random((*shape, n)), axis=-1)[..., :k] + + def actions(self, probs: np.ndarray) -> np.ndarray: + """Per opponent state, the share of its weight each action carries: σ itself, or k draws from σ at 1/k each.""" + k = self.sampling.opponent_actions + if k is None: + return probs + cdf = np.cumsum(probs, axis=-1) + cdf[..., -1] = 1.0 # a row summing to 1 − ulp must still pick an action + columns = np.arange(probs.shape[-1]) + counts = np.zeros_like(probs) + for _ in range(k): + pick = (self.rng.random(probs.shape[:-1])[..., None] < cdf).argmax(axis=-1) + counts += pick[..., None] == columns + return counts / k + + def opponents(self, u: np.ndarray, clash: np.ndarray) -> np.ndarray: + """Per entry, the opponent's weighted vector, or k hands drawn from its live weight at 1/k of the total each.""" + k = self.sampling.opponent_range + if k is None: + return u + live = np.where(clash, 0.0, u) + totals = live.sum(axis=1) + out = np.zeros_like(u) + for e, total in enumerate(totals): + if total > 0.0: + picks = self.rng.choice(u.shape[1], size=k, p=live[e] / total) + out[e] = np.bincount(picks, minlength=u.shape[1]) * (total / k) + return out + +_EXACT_DRAWS = _Draws(EXACT, None) + +# --------------------------------------------------------------------------- +# The frame: physical states and board rows +# --------------------------------------------------------------------------- + +HOLES: tuple[Hole, ...] = tuple(combinations(DECK, HOLE_CARDS)) +SLOTS = 1 + len(DECK) +N_STATES = len(HOLES) * SLOTS + +_HOLE_INDEX = {hole: index for index, hole in enumerate(HOLES)} +_BOARD_SETS = tuple(combinations(DECK, 2)) +_SET_INDEX = {pair: index for index, pair in enumerate(_BOARD_SETS)} +_LEVEL_ROWS = (1, len(DECK), len(DECK) * (len(DECK) - 1)) + +def state_index(hole: Hole, thrown: Hole) -> int: + """The row of `(hole, thrown)` in a seat's vector; `thrown` is empty or one card.""" + slot = 0 if not thrown else 1 + DECK.index(thrown[0]) + return _HOLE_INDEX[tuple(sorted(hole))] * SLOTS + slot + +def board_row(board: tuple[Card, ...]) -> int: + """A board's batch row: deck order for one card, `(first, second)` ordered for two. + + Ordered pairs, 210 rows rather than 105 sets, because round 1 was played + against the first card alone and the two orders carry different reach + into round 2; only the showdown, which cannot tell them apart, is per set. + """ + if len(board) == 1: + return DECK.index(board[0]) + first, second = DECK.index(board[0]), DECK.index(board[1]) + return first * (len(DECK) - 1) + (second if second < first else second - 1) + +def _state_cards() -> tuple[np.ndarray, np.ndarray]: + """Per state: which deck cards it accounts for (hole and thrown), and whether it is a real state.""" + holds = np.zeros((N_STATES, len(DECK)), dtype=bool) + valid = np.zeros(N_STATES, dtype=bool) + for h, hole in enumerate(HOLES): + for slot in range(SLOTS): + s = h * SLOTS + slot + for card in hole: + holds[s, DECK.index(card)] = True + if slot == 0: + valid[s] = True + elif not holds[s, slot - 1]: + holds[s, slot - 1] = True + valid[s] = True + return holds, valid + +# --------------------------------------------------------------------------- +# The description: levels, key maps, draw maps, the public tree, the showdown ranks +# --------------------------------------------------------------------------- + +@dataclass(frozen=True, slots=True, eq=False) +class _Level: + """The boards at one board count: `ok[b, s]` is 1 when state s avoids board b; `parent[b]` is the board above b and `children_of[p]` the boards of this level under p.""" + + ok: np.ndarray + parent: np.ndarray + children_of: np.ndarray + +@dataclass(frozen=True, slots=True, eq=False) +class _KeyMap: + """(board row, state) → position in one shape's `private_keys`, −1 outside its support; and the inverse.""" + + index: np.ndarray # int32 (n_boards, N_STATES) + n_keys: int + entry_start: np.ndarray # int64 (n_keys + 1,): entries of key k are entry_*[start[k]:start[k+1]] + entry_board: np.ndarray # int32 + entry_state: np.ndarray # int32 + +@dataclass(frozen=True, slots=True, eq=False) +class _DrawMap: + """One throw as a map from pre-draw holes to post-draw states: `targets[b, k, j]` on board b, hole k, replacement j.""" + + source: np.ndarray # int32 (455,): the pre-draw state of each hole + targets: np.ndarray # int32 (15, 455, 12) + +@dataclass(frozen=True, slots=True, eq=False) +class _Edge: + """The actions of a decision that lead to one child, with the draw map of each throw (None for a bet or standing pat).""" + + child: _Node + actions: tuple[int, ...] + draws: tuple[_DrawMap | None, ...] + +@dataclass(frozen=True, slots=True, eq=False) +class _Node: + """One public node. `point` is a decision's index among the 141 table points; `deals_board` tells a board deal from a replacement.""" + + index: int + kind: NodeKind + level: int + discards: tuple[int, int] + player: int | None + actions: tuple[Any, ...] + point: int | None + edges: tuple[_Edge, ...] + child: _Node | None + prob: float + deals_board: bool + fold_winner: int | None + stake: float + pot: float + committed: tuple[float, float] + +@dataclass(frozen=True, slots=True, eq=False) +class _Description: + """Everything that does not depend on the deal set, built once per process.""" + + points: tuple[PublicPoint, ...] + row_offset: np.ndarray + column_offset: np.ndarray + width: np.ndarray + holds: np.ndarray # bool (N_STATES, 15) + valid: np.ndarray # bool (N_STATES,) + hole_of: np.ndarray # int32 (N_STATES,) + hole_holds: np.ndarray # float64 (455, 15): which cards each hole holds + levels: tuple[_Level, _Level, _Level] + set_of_row: np.ndarray # int32 (210,): level-2 row → board set, for the showdown alone + keys: dict[tuple[int, int], _KeyMap] + root: _Node + parent: dict[int, tuple[_Node, _Edge | None]] + decision_of_point: tuple[_Node, ...] + inner: np.ndarray # int32 (455,): dense rank of each hole's inner hand + outer: np.ndarray # int32 (105, 455): dense rank of each hole's outer hand with each board set, 0 where they clash + +def _levels(holds: np.ndarray, valid: np.ndarray) -> tuple[_Level, _Level, _Level]: + n = len(DECK) + # `parent` and `children_of` both live on the CHILD level: `parent[b]` is + # the board above b, `children_of[p]` every board of this level under p. + zero = _Level( + ok=valid[None, :].astype(np.float64), + parent=np.zeros(1, np.int32), + children_of=np.zeros((0, 0), np.int32), + ) + one = _Level( + ok=(valid[None, :] & ~holds.T).astype(np.float64), + parent=np.zeros(n, np.int32), + children_of=np.arange(n, dtype=np.int32)[None, :], + ) + ok = np.zeros((n * (n - 1), N_STATES)) + parent = np.zeros(n * (n - 1), np.int32) + for first in range(n): + for second in range(n): + if first != second: + row = board_row((DECK[first], DECK[second])) + ok[row] = valid & ~holds[:, first] & ~holds[:, second] + parent[row] = first + two = _Level( + ok=ok, + parent=parent, + children_of=np.stack([np.flatnonzero(parent == first) for first in range(n)]).astype(np.int32), + ) + return zero, one, two + +def _key_map(level: int, discards: int) -> _KeyMap: + """(board row, state) → position in `private_keys(level, discards)`, built per ORDERED board row, and its inverse.""" + keys = private_keys(board_cards=level, discards=discards) + position = {key: k for k, key in enumerate(keys)} + boards = [(card,) for card in DECK] if level == 1 else list(permutations(DECK, 2)) + index = np.full((_LEVEL_ROWS[level], N_STATES), -1, np.int32) + for board in boards: + b = board_row(board) + for h, hole in enumerate(HOLES): + if set(hole) & set(board): + continue + if discards == 0: + index[b, h * SLOTS] = position[canonical_picture(hole, (), board)] + continue + for card in DECK: + if card in hole or card in board: + continue + index[b, h * SLOTS + 1 + DECK.index(card)] = position[ + canonical_picture(hole, (card,), board) + ] + flat = index.ravel() + order = np.argsort(flat, kind="stable") + order = order[flat[order] >= 0] + sorted_keys = flat[order] + start = np.searchsorted(sorted_keys, np.arange(len(keys) + 1)).astype(np.int64) + return _KeyMap( + index=index, + n_keys=len(keys), + entry_start=start, + entry_board=(order // N_STATES).astype(np.int32), + entry_state=(order % N_STATES).astype(np.int32), + ) + +def _draw_maps() -> tuple[_DrawMap, _DrawMap, _DrawMap]: + """The three one-card throws as maps, one board card out, every replacement listed. + + Which physical card `THROW_LOW` names depends on the board through the + canonical relabelling, so the thrown card is read off `draw_order`. + """ + n = len(DECK) + replacements = n - HOLE_CARDS + targets = np.zeros((n, len(HOLES), HOLE_CARDS, replacements), np.int32) + for b, board_card in enumerate(DECK): + for h, hole in enumerate(HOLES): + ordered = draw_order(hole=hole, discarded=(), board=(board_card,)) + for k, thrown in enumerate(ordered): + kept = tuple(card for card in hole if card != thrown) + spare = [card for card in DECK if card not in hole] + for j, drawn in enumerate(spare): + targets[b, h, k, j] = state_index(kept + (drawn,), (thrown,)) + source = (np.arange(len(HOLES)) * SLOTS).astype(np.int32) + return tuple(_DrawMap(source=source, targets=targets[:, :, k, :]) for k in range(HOLE_CARDS)) + +def _public_tree( + draws: tuple[_DrawMap, ...], points: tuple[PublicPoint, ...] +) -> tuple[_Node, dict[int, tuple[_Node, _Edge | None]], tuple[_Node, ...]]: + """The public tree with edges and parent links, walked from one representative deal. + + Replacement deals are told apart from board deals (the census merges + them), and a decision is tied to its table point by public signature. + """ + by_signature = { + (point.board_cards, point.draws, point.betting): index for index, point in enumerate(points) + } + root_state = MiniState(holes=(DECK[0:HOLE_CARDS], DECK[HOLE_CARDS : 2 * HOLE_CARDS])) + nodes: dict[tuple, _Node] = {} + order: list[_Node | None] = [] + parent: dict[int, tuple[_Node, _Edge | None]] = {} + decision_of_point: dict[int, _Node] = {} + + def visit(state: MiniState) -> _Node: + signature = ( + state.kind, + len(state.board), + state.draws, + state.betting, + tuple(len(hole) for hole in state.holes), + ) + if signature in nodes: + return nodes[signature] + index = len(order) + order.append(None) + fields: dict[str, Any] = dict( + index=index, + kind=state.kind, + level=len(state.board), + discards=tuple(len(pile) for pile in state.discards), + player=None, + actions=(), + point=None, + edges=(), + child=None, + prob=0.0, + deals_board=False, + fold_winner=None, + stake=0.0, + pot=0.0, + committed=(0.0, 0.0), + ) + if state.kind is NodeKind.TERMINAL: + chips = chip_state(state.betting) + line = state.betting[-1] + if line and line[-1] is Action.FOLD: + loser = (len(line) - 1) % 2 + fields.update(fold_winner=1 - loser, stake=float(chips.committed[loser])) + else: + fields.update( + pot=float(chips.pot), + committed=(float(chips.committed[0]), float(chips.committed[1])), + ) + node = _Node(**fields) + elif state.kind is NodeKind.CHANCE: + outcomes = state.chance_outcomes() + child = visit(state.apply_chance(outcomes[0][0])) + fields.update( + child=child, + prob=outcomes[0][1], + # A short hole means the deck owes a replacement; otherwise a board card. + deals_board=all(len(hole) == HOLE_CARDS for hole in state.holes), + ) + node = _Node(**fields) + parent[child.index] = (node, None) + else: + actions = state.legal_actions() + grouped: dict[int, tuple[list[int], list[_DrawMap | None]]] = {} + children: dict[int, _Node] = {} + for k, action in enumerate(actions): + child = visit(state.apply(action)) + children[child.index] = child + columns, maps = grouped.setdefault(child.index, ([], [])) + columns.append(k) + maps.append( + draws[DRAW_ACTIONS.index(action) - 1] + if state.is_draw_decision() and throw_count(action) > 0 + else None + ) + edges = tuple( + _Edge(child=children[child_index], actions=tuple(columns), draws=tuple(maps)) + for child_index, (columns, maps) in grouped.items() + ) + fields.update( + player=state.current_player, + actions=actions, + point=by_signature[(len(state.board), state.draws, state.betting)], + edges=edges, + ) + node = _Node(**fields) + for edge in edges: + parent[edge.child.index] = (node, edge) + decision_of_point[node.point] = node + nodes[signature] = node + order[index] = node + return node + + root = visit(root_state) + if len(decision_of_point) != len(points): + raise AssertionError("the public tree and the enumerator disagree on the decision points") + return root, parent, tuple(decision_of_point[p] for p in range(len(points))) + +def _dense_rank(scores: Sequence[Any]) -> np.ndarray: + """Scores as integers with the same order, so a sign of a difference is a comparison.""" + ladder = {score: rank for rank, score in enumerate(sorted(set(scores)))} + return np.array([ladder[score] for score in scores], dtype=np.int32) + +def _showdown_ranks() -> tuple[np.ndarray, np.ndarray]: + """Per hole, the dense rank of its inner hand, and of its outer hand with each board set (0 where the hole clashes).""" + inner = _dense_rank([inner_score(hole) for hole in HOLES]) + outer = np.zeros((len(_BOARD_SETS), len(HOLES)), np.int32) + for s, board in enumerate(_BOARD_SETS): + live = [h for h, hole in enumerate(HOLES) if not set(hole) & set(board)] + outer[s, live] = _dense_rank([outer_score(HOLES[h], board) for h in live]) + return inner, outer + +@cache +def _description() -> _Description: + points = public_decision_points() + sizes = [len(private_keys(board_cards=p.board_cards, discards=p.discards)) for p in points] + widths = [p.width for p in points] + holds, valid = _state_cards() + hole_of = (np.arange(N_STATES) // SLOTS).astype(np.int32) + set_of_row = np.zeros(_LEVEL_ROWS[2], np.int32) + for first in DECK: + for second in DECK: + if first != second: + set_of_row[board_row((first, second))] = _SET_INDEX[tuple(sorted((first, second)))] + root, parent, decision_of_point = _public_tree(_draw_maps(), points) + inner, outer = _showdown_ranks() + return _Description( + points=points, + row_offset=np.concatenate([[0], np.cumsum(sizes)]).astype(np.int64), + column_offset=np.concatenate( + [[0], np.cumsum([s * w for s, w in zip(sizes, widths, strict=True)])] + ).astype(np.int64), + width=np.asarray(widths, dtype=np.int64), + holds=holds, + valid=valid, + hole_of=hole_of, + hole_holds=holds[::SLOTS].astype(np.float64), + levels=_levels(holds, valid), + set_of_row=set_of_row, + keys={shape: _key_map(*shape) for shape in ((1, 0), (2, 0), (2, 1))}, + root=root, + parent=parent, + decision_of_point=decision_of_point, + inner=inner, + outer=outer, + ) + +# --------------------------------------------------------------------------- +# The deal set +# --------------------------------------------------------------------------- + +def _weights(holes: tuple[Iterable[Hole], Iterable[Hole]] | None) -> tuple[np.ndarray, np.ndarray]: + """Per-seat deal probability at pre-draw states, zero elsewhere: uniform over the allowed disjoint pairs. + + P(r, h) = weight[0][r] · weight[1][h] · [r ∩ h = ∅]; 1/100,100 in the + full game, carried by seat 0 with seat 1 at 1 — the walk's `random_deal` + distribution, and the grader's convention. + """ + allowed = ( + (set(HOLES), set(HOLES)) + if holes is None + else (set(map(tuple, map(sorted, holes[0]))), set(map(tuple, map(sorted, holes[1])))) + ) + for seat, chosen in enumerate(allowed): + unknown = chosen - set(HOLES) + if unknown: + raise ValueError(f"seat {seat} cannot be dealt {sorted(unknown)}") + pairs = sum(1 for r in allowed[0] for h in allowed[1] if not set(r) & set(h)) + if pairs == 0: + raise ValueError("no deal gives both seats a hole from their allowed sets") + out = [] + for seat, chosen in enumerate(allowed): + weight = np.zeros(N_STATES) + weight[::SLOTS] = [1.0 if hole in chosen else 0.0 for hole in HOLES] + if seat == 0: + weight /= pairs + out.append(weight) + return out[0], out[1] + +def _own_deal(weight: np.ndarray) -> dict[tuple[int, int], np.ndarray]: + """The traverser's own deal factor per state, per shape: the pre-draw weight, or the sum over the three pre-draw holes of a post-draw state.""" + post = np.zeros(N_STATES) + for h, hole in enumerate(HOLES): + for slot in range(1, SLOTS): + thrown = DECK[slot - 1] + if thrown in hole: + continue + post[h * SLOTS + slot] = sum( + weight[state_index(tuple(c for c in hole if c != drawn) + (thrown,), ())] + for drawn in hole + ) + return {(1, 0): weight, (2, 0): weight, (2, 1): post} + +# --------------------------------------------------------------------------- +# The targets +# --------------------------------------------------------------------------- + +_Sigma = Sequence[np.ndarray] + +@dataclass(frozen=True, eq=False) +class BoostTargets: + """Sampled (or exact) regret and exact arrival for any table row, under a strategy read from the materialised table. + + boost = BoostTargets.build() # once per process + sigma = boost.sigma(table) # the walk's σ, per point + regret, arrival = boost.targets(sigma, seat=0, rows=rows, sampling=Sampling(), seed=seed, t=t) + regret, arrival = boost.targets(sigma, seat=0, rows=rows, sampling=EXACT) # the oracle + + `regret` is `(n, 4)` float64, zero past each row's width; `arrival` is + `(n,)`, the expected number of visits one walk with traverser `seat` + pays the row (a probability, but for the rare walk that reaches a row + twice through two throws). + `arrivals` is the second number alone, at a fraction of the cost. + `build(holes=)` restricts what each seat may be dealt, uniformly over + the pairs that remain, as `exploitability.minidrawmaha_game(holes=)` does + and as a test's `deal` must then sample; the default is every deal. + """ + + description: _Description + weight: tuple[np.ndarray, np.ndarray] + own_deal: tuple[dict[tuple[int, int], np.ndarray], dict[tuple[int, int], np.ndarray]] + + @classmethod + def build(cls, holes: tuple[Iterable[Hole], Iterable[Hole]] | None = None) -> BoostTargets: + """The description (shared, built once) with this deal set's weights. The default is cached.""" + if holes is None: + return _full_game() + weights = _weights(holes) + return cls(_description(), weights, (_own_deal(weights[0]), _own_deal(weights[1]))) + + @property + def points(self) -> tuple[PublicPoint, ...]: + return self.description.points + + def entries_of(self, shape: tuple[int, int], position: int) -> tuple[np.ndarray, np.ndarray]: + """The physical (board row, state) entries of key `position` in `shape`'s `private_keys`.""" + keys = self.description.keys[shape] + span = slice(keys.entry_start[position], keys.entry_start[position + 1]) + return keys.entry_board[span], keys.entry_state[span] + + # -- σ ------------------------------------------------------------------ + + def sigma(self, table: PackedTable) -> tuple[np.ndarray, ...]: + """Regret matching over the whole-game table, one `(n_keys, width)` array per point, float64. + + The positive part of each row over its total, uniform where nothing + is positive: `RegretMatcher.strategy`, vectorised with one `reduceat`, + which adds a row left to right where the ledger's `sum` may pair, so + a row can differ from the walk's by one ulp. Refuses a table that is + not the whole game in the whole game's order. + """ + d = self.description + widths = table.widths() + if ( + len(table) != d.row_offset[-1] + or table.cumulative_regret.size != d.column_offset[-1] + or not np.array_equal(widths, np.repeat(d.width, np.diff(d.row_offset))) + ): + raise ValueError( + "the exact target reads the whole game's layout and was handed a table of " + f"{len(table):,} rows that is not the whole game in the whole game's order" + ) + positive = np.maximum(table.cumulative_regret, 0.0) + starts = np.cumsum(widths) - widths + totals = np.add.reduceat(positive, starts) + per_column = np.repeat(totals, widths) + uniform = np.repeat(1.0 / widths, widths) + with np.errstate(invalid="ignore", divide="ignore"): + flat = np.where(per_column > 0.0, positive / per_column, uniform) + return tuple( + flat[d.column_offset[p] : d.column_offset[p + 1]].reshape(-1, int(d.width[p])) + for p in range(len(d.points)) + ) + + # -- the targets -------------------------------------------------------- + + def targets( + self, + sigma: _Sigma, + *, + seat: int, + rows: np.ndarray, + sampling: Sampling, + seed: int | None = None, + t: int | None = None, + ) -> tuple[np.ndarray, np.ndarray]: + """Regret `(n, 4)` and arrival `(n,)` — expected visits per walk — for table `rows`, all of them `seat`'s, under `sigma`. + + The regret is exact under `EXACT` and an unbiased estimate of it + otherwise; the arrival is exact under any `sampling`. A sampled call + names `seed` and `t`, and row r draws from + `default_rng([seed, t, seat, r, _STREAM_TAG])` alone, so its target + does not depend on which rows share the call or in what order — and + a row asked for twice gets the same sample twice: more samples of one + row are `playouts`, not repeats. A row the + walk cannot reach (zero arrival) gets zero regret, not NaN; a row of + the other seat raises, as does one outside the table. + """ + if not isinstance(sampling, Sampling): + raise TypeError(f"sampling must be a Sampling (EXACT for the exact sum), got {sampling!r}") + if not sampling.is_exact: + for name, value in (("seed", seed), ("t", t)): + if isinstance(value, bool) or not isinstance(value, numbers.Integral) or value < 0: + raise ValueError(f"a sampled target needs a non-negative int {name}, got {value!r}") + seed, t = int(seed), int(t) + rows = self._rows(seat, rows) + d = self.description + regret = np.zeros((rows.size, TARGET_WIDTH)) + arrival = np.zeros(rows.size) + memo: dict[tuple[int, int, int], np.ndarray] = {} + for i, (node, position, boards, states, u, total) in enumerate( + self._entries(sigma, seat, rows, memo) + ): + arrival[i] = total + if total <= 0.0: + continue + draws = ( + _EXACT_DRAWS + if sampling.is_exact + else _Draws(sampling, np.random.default_rng([seed, t, seat, int(rows[i]), _STREAM_TAG])) + ) + # Opponent states sharing a card with the entry's hero state; only + # a range draw needs them, since the play-out's compat masks them. + clash = ( + None + if sampling.opponent_range is None + else (d.holds.astype(np.uint8) @ d.holds[states].T.astype(np.uint8)).T > 0 + ) + mix = sigma[node.point][position] + summed = np.zeros(len(mix)) + for _ in range(sampling.playouts): + facing = u if clash is None else draws.opponents(u, clash) + values = self._action_values(sigma, seat, node, facing, states[:, None], boards, draws)[:, 0, :] + summed += (values - (values @ mix)[:, None]).sum(axis=0) + regret[i, : len(mix)] = summed / (sampling.playouts * total) + return regret, arrival + + def arrivals(self, sigma: _Sigma, *, seat: int, rows: np.ndarray) -> np.ndarray: + """The arrival alone for table `rows`: the reach and the compatibility sum, no value below.""" + rows = self._rows(seat, rows) + memo: dict[tuple[int, int, int], np.ndarray] = {} + return np.fromiter( + (total for *_, total in self._entries(sigma, seat, rows, memo)), dtype=np.float64, count=rows.size + ) + + def reach(self, sigma: _Sigma, *, carried: int, point: int, board: int) -> np.ndarray: + """Seat `carried`'s reach at `point` on one board, over its physical states: deal weight × chance × its own σ along the path. + + What the grader carries down for the responder's opponent; here the + row's opponent is `carried = 1 − seat`. Hero's actions never enter. + """ + if carried not in SEATS: + raise ValueError(f"carried must be one of {SEATS}, got {carried!r}") + # A copy: at the root the vector is the deal weight itself. + return self._reach(sigma, carried, self.description.decision_of_point[point], board, {}).copy() + + def _rows(self, seat: int, rows: np.ndarray) -> np.ndarray: + if seat not in SEATS: + raise ValueError(f"seat must be one of {SEATS}, got {seat!r}") + d = self.description + rows = np.asarray(rows, dtype=np.int64) + if rows.ndim != 1: + raise ValueError(f"rows must be a 1-D array of table rows, got shape {rows.shape}") + if rows.size and (rows.min() < 0 or rows.max() >= d.row_offset[-1]): + raise IndexError(f"a row is outside the table's {d.row_offset[-1]:,} rows") + return rows + + def _entries(self, sigma: _Sigma, seat: int, rows: np.ndarray, memo: dict): + """Per row: its node and position, its physical entries, the weighted opponent vector at each, and the arrival.""" + d = self.description + own = self.own_deal[seat] + for row in rows.tolist(): + p = int(np.searchsorted(d.row_offset, row, side="right") - 1) + position = row - int(d.row_offset[p]) + node = d.decision_of_point[p] + if node.player != seat: + raise ValueError(f"row {row} belongs to seat {node.player}, not seat {seat}") + shape = (node.level, node.discards[seat]) + boards, states = self.entries_of(shape, position) + reach = np.stack([self._reach(sigma, 1 - seat, node, int(b), memo) for b in boards]) + u = own[shape][states][:, None] * reach + compat, _ = self._showdown(node.level, boards, states[:, None], u, shares=False) + yield node, position, boards, states, u, float(compat.sum()) + + # -- the back-solve ---------------------------------------------------- + + def _reach(self, sigma: _Sigma, carried: int, node: _Node, b: int, memo: dict) -> np.ndarray: + d = self.description + key = (carried, node.index, b) + if key in memo: + return memo[key] + if node is d.root: + return self.weight[carried] + above, edge = d.parent[node.index] + if above.kind is NodeKind.CHANCE: + level = d.levels[node.level] + if above.deals_board: + base = self._reach(sigma, carried, above, int(level.parent[b]), memo) + else: + base = self._reach(sigma, carried, above, b, memo) + out = above.prob * base * level.ok[b] + elif above.player != carried: + out = self._reach(sigma, carried, above, b, memo) + else: + base = self._reach(sigma, carried, above, b, memo) + index = d.keys[(above.level, above.discards[carried])].index[b] + valid = index >= 0 + safe = np.where(valid, index, 0) + table = sigma[above.point] + out = np.zeros(N_STATES) + for column, draw in zip(edge.actions, edge.draws, strict=True): + played = base * np.where(valid, table[safe, column], 0.0) + if draw is None: + out += played + else: + out += _push(played[None, :], draw, np.array([b]))[0] + # Memoised through the board-2 deal: the 66 level-1 nodes on the + # paths (decisions and chance nodes) have 15 boards each and the 20 + # round-2 opening decisions 210 each; the rest of round 2 is at most + # two opponent steps away and is recomputed, so a call's memo is at + # most 5,190 vectors whatever rows it is handed. + if node.level <= 1 or (above.kind is NodeKind.CHANCE and above.deals_board): + memo[key] = out + return out + + # -- the value below ---------------------------------------------------- + # + # The recursion carries, per board b_i: the opponent's weighted vector + # u_i (over its 7,280 states) and a SET of hero states h_i (the same + # count on every board). Hero's own moves never touch u; a throw widens + # the set twelvefold per throw, the board-2 deal widens the boards + # fourteenfold, and values come back as one number per (board, hero + # state). Keeping u per board rather than per hero state is what makes a + # round-1 row affordable: the opponent's σ gathers and pushes below are + # done once per board, not once per hero continuation. + + def _action_values( + self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray, draws: _Draws + ) -> np.ndarray: + """Hero's weighted value below each action of `node`: `(nb, nh, width)` for boards b, hero states h (nb, nh), opponent vectors u (nb, S).""" + d = self.description + nb, nh = h.shape + values = np.zeros((nb, nh, len(node.actions))) + for edge in node.edges: + if all(draw is None for draw in edge.draws): + below = self._value(sigma, hero, edge.child, u, h, b, draws) + for column in edge.actions: + values[..., column] = below + continue + # The throw edge: the three throws' replacement states, stacked, + # walk the one subtree below in one pass and are split after. + # Under sampling each (board, hero state) keeps k of its twelve + # replacements per throw, scaled 12/k; hero's throws themselves + # are all walked. + hole = d.hole_of[h] + fan = edge.draws[0].targets.shape[-1] + parts = [] + for draw in edge.draws: + replacements = draw.targets[b[:, None], hole] # (nb, nh, 12) + pick = draws.subset(draws.sampling.hero_draws, fan, (nb, nh)) + parts.append(replacements if pick is None else np.take_along_axis(replacements, pick, axis=-1)) + kept = parts[0].shape[-1] + stacked = np.concatenate(parts, axis=-1) # (nb, nh, 3 × kept) + below = self._value(sigma, hero, edge.child, u, stacked.reshape(nb, -1), b, draws) + below = below.reshape(nb, nh, len(edge.draws), kept).sum(axis=-1) + if kept < fan: + below *= fan / kept + for j, column in enumerate(edge.actions): + values[..., column] = below[..., j] + return values + + def _value( + self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray, draws: _Draws + ) -> np.ndarray: + """Hero's weighted expected chips from `node`, `(nb, nh)`, both seats at σ, the deck averaged (or sampled, by `draws`).""" + d = self.description + nb, nh = h.shape + if node.kind is NodeKind.TERMINAL: + compat, share = self._showdown(node.level, b, h, u, shares=node.fold_winner is None) + if node.fold_winner is not None: + sign = 1.0 if node.fold_winner == hero else -1.0 + return sign * node.stake * compat + return node.pot * share - node.committed[hero] * compat + if node.kind is NodeKind.CHANCE: + if node.deals_board: + level = d.levels[node.child.level] + kids = level.children_of[b] # (nb, 14) + full = kids.shape[1] + pick = draws.subset(draws.sampling.board_cards, full, (nb,)) + if pick is not None: + kids = np.take_along_axis(kids, pick, axis=1) + fan = kids.shape[1] + spread = node.prob * u[:, None, :] * level.ok[kids] + if fan < full: + spread *= full / fan + below = self._value( + sigma, hero, node.child, + spread.reshape(-1, N_STATES), np.repeat(h, fan, axis=0), kids.ravel(), draws, + ) + return below.reshape(nb, fan, nh).sum(axis=1) + return self._value( + sigma, hero, node.child, node.prob * u * d.levels[node.level].ok[b], h, b, draws + ) + if node.player == hero: + index = d.keys[(node.level, node.discards[hero])].index[b[:, None], h] # (nb, nh) + valid = index >= 0 + mix = np.where(valid[..., None], sigma[node.point][np.where(valid, index, 0)], 0.0) + return (self._action_values(sigma, hero, node, u, h, b, draws) * mix).sum(axis=-1) + index = d.keys[(node.level, node.discards[node.player])].index[b] # (nb, S) + valid = index >= 0 + safe = np.where(valid, index, 0) + share = draws.actions(sigma[node.point][safe]) # (nb, S, width) + total = np.zeros((nb, nh)) + for edge in node.edges: + pushed = np.zeros_like(u) + for column, draw in zip(edge.actions, edge.draws, strict=True): + played = u * np.where(valid, share[..., column], 0.0) + if draw is None: + pushed += played + continue + pick = draws.subset(draws.sampling.opponent_draws, draw.targets.shape[-1], (nb, len(HOLES))) + pushed += _push(played, draw, b, pick) + # A line no opponent state takes is worth nothing below; under + # action sampling most are, and walking them would cost the + # whole subtree for a zero. + if pushed.any(): + total += self._value(sigma, hero, edge.child, pushed, h, b, draws) + return total + + # -- the showdown and compatibility ------------------------------------ + + def _showdown( + self, level: int, b: np.ndarray, h: np.ndarray, u: np.ndarray, *, shares: bool + ) -> tuple[np.ndarray, np.ndarray | None]: + """Per (board i, hero state j): Σ_o u_i[o]·compat(h_ij, o) and, with `shares`, the same weighted by hero's pot share. + + Compatibility is the two card sets being disjoint — hero's hole and + thrown card against the opponent's hole and thrown card — and a hero + state that holds a card of its own board is no state and scores + zero. Folded over the opponent's HOLES: u_i summed over the 16 slots + of each hole gives the hole's total reach, and the slots of the cards + hero holds are subtracted back out, because an opponent who threw one + of hero's cards cannot be there. Shares come from the dense ranks of + both halves: ¼(2 + sign(inner) + sign(outer)). + """ + d = self.description + nb, nh = h.shape + folded = u.reshape(nb, len(HOLES), SLOTS) + total = folded.sum(axis=-1) # (nb, 455) + mine = d.holds[h].astype(np.float64).transpose(0, 2, 1) # (nb, 15, nh) + shared = d.hole_holds @ mine # (nb, 455, nh): cards an opponent hole shares with hero + excluded = folded[:, :, 1:] @ mine # (nb, 455, nh): reach of that hole's states that threw a card of hero's + terms = np.where(shared == 0.0, total[:, :, None] - excluded, 0.0) + live = (d.valid[h] & (d.levels[level].ok[b[:, None], h] > 0.0)).astype(np.float64) + compat = terms.sum(axis=1) * live + if not shares: + return compat, None + hole = d.hole_of[h] # (nb, nh) + sets = d.set_of_row[b] # (nb,) + inner = np.sign(d.inner[hole][:, None, :] - d.inner[None, :, None]) + outer = np.sign(d.outer[sets[:, None], hole][:, None, :] - d.outer[sets][:, :, None]) + share = 0.25 * (2.0 + inner + outer) + return compat, (terms * share).sum(axis=1) * live + +def _push(played: np.ndarray, draw: _DrawMap, b: np.ndarray, pick: np.ndarray | None = None) -> np.ndarray: + """Each pair's vector through a throw on its level-1 board: every pre-draw hole spreads over its twelve replacements, or over the k of them `pick` keeps, scaled 12/k.""" + m = played.shape[0] + targets = draw.targets[b] # (m, 455, 12) + full = targets.shape[2] + if pick is not None: + targets = np.take_along_axis(targets, pick, axis=2) + offsets = (np.arange(m) * N_STATES)[:, None, None] + weights = np.repeat(played[:, draw.source], targets.shape[2], axis=1).ravel() + if targets.shape[2] < full: + weights *= full / targets.shape[2] + return np.bincount((offsets + targets).ravel(), weights=weights, minlength=m * N_STATES).reshape( + m, N_STATES + ) + +@cache +def _full_game() -> BoostTargets: + weights = _weights(None) + return BoostTargets(_description(), weights, (_own_deal(weights[0]), _own_deal(weights[1]))) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 84fe5be7..fd5c973e 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -25,7 +25,8 @@ loop. them. Loosening one to make a test pass is a finding to report, not a fix. - **Slow and heavy work is gated by an environment variable named in the skip reason**: `MINIDRAWMAHA_FULL_TABLE`, `MINIDRAWMAHA_FULL_GRADE`, - `MINIDRAWMAHA_FULL_CALIBRATION`, `MINIDRAWMAHA_FULL_DEEP`. Tests that need + `MINIDRAWMAHA_FULL_CALIBRATION`, `MINIDRAWMAHA_FULL_DEEP`, + `MINIDRAWMAHA_FULL_TARGET`. Tests that need torch `importorskip` it. The default suite must keep passing without torch, Modal, OpenSpiel or a network download. - **Committed numbers are recomputed, not transcribed.** `minidrawmaha/test_play.py` diff --git a/tests/deepcfr/test_fit.py b/tests/deepcfr/test_fit.py index 431cb957..e5ada02c 100644 --- a/tests/deepcfr/test_fit.py +++ b/tests/deepcfr/test_fit.py @@ -101,6 +101,91 @@ def test_later_iterations_weigh_more(): ) np.testing.assert_allclose(legal_prediction(net, row), 0.5, atol=0.05) +def test_weight_one_is_the_loss_before_the_column_existed_bit_for_bit(): + # The loss as it was written before the weight column — the squared + # error over the legal columns, each row scaled by t / mean(t) — restated + # here so the pin is against the formula, not against the code under + # test. t · 1.0 is t exactly in float32, so the two agree to the bit. + rows, t, targets = real_rows(300, seed=4) + torch.manual_seed(0) + net = Net() + built = layout() + point, _ = built.locate_rows(rows) + gather = torch.from_numpy(legal_gather()[point].astype(np.int64)) + legal = gather != NO_ACTION + with torch.no_grad(): + outputs = net(torch.from_numpy(built.features(rows))) + predicted = outputs.gather(1, gather.clamp(min=0)) + target = torch.where(legal, torch.from_numpy(targets), 0.0) + error = torch.where(legal, (predicted - target) ** 2, 0.0).sum(dim=1) + t32 = torch.from_numpy(t.astype(np.float32)) + before = ((t32 / t32.mean()) * error).mean() + now = batch_loss(net, rows, t, targets, np.ones(300, np.float32)) + default = batch_loss(net, rows, t, targets) + assert torch.equal(now, before) and torch.equal(default, before) + # And through the whole fit: the weights a weight-one fit lands on are + # the ones a fit told nothing about weights lands on. + plain = fit_from_scratch(rows, t, targets, seed=(0, 1, 0), config=SMALL) + ones = fit_from_scratch(rows, t, targets, np.ones(300, np.float32), seed=(0, 1, 0), config=SMALL) + for a, b in zip(plain.parameters(), ones.parameters(), strict=True): + assert torch.equal(a, b) + +def test_a_row_of_weight_zero_is_inert(): + # One row twice at t = 1: +1 at weight 0 and −1 at weight 1. The first + # row must leave no trace, so the fit lands on −1, not on the mean 0. + (row,), _, _ = real_rows(1, seed=3) + rows = np.array([row, row]) + t = np.array([1, 1]) + targets = np.array([[1, 1, 1, 1], [-1, -1, -1, -1]], dtype=np.float32) + net = fit_from_scratch( + rows, t, targets, np.array([0.0, 1.0]), seed=(0, 1, 0), config=FitConfig(steps=1_500, batch=64, threads=1) + ) + np.testing.assert_allclose(legal_prediction(net, row), -1.0, atol=0.05) + +def test_the_loss_weighs_a_row_by_t_times_weight(): + # The same two rows as `test_later_iterations_weigh_more`, +1 at t = 3 and + # −1 at t = 1, with weights (1, 3): t · w is 3 for both, so the weighted + # least-squares answer moves from +0.5 to the plain mean, 0. + (row,), _, _ = real_rows(1, seed=3) + rows = np.array([row, row]) + t = np.array([3, 1]) + targets = np.array([[1, 1, 1, 1], [-1, -1, -1, -1]], dtype=np.float32) + net = fit_from_scratch( + rows, t, targets, np.array([1.0, 3.0]), seed=(0, 1, 0), config=FitConfig(steps=1_500, batch=64, threads=1) + ) + np.testing.assert_allclose(legal_prediction(net, row), 0.0, atol=0.05) + +def test_a_batch_of_weightless_rows_has_a_finite_loss(): + # Every row of the batch at weight 0 (a tiny memory under the boost's α = 0): + # the normalising mean is 0, and 0/0 would make the loss NaN and the fit + # "diverged". The loss is 0 instead, and its gradient finite. + rows, t, targets = real_rows(16, seed=8) + torch.manual_seed(0) + net = Net() + loss = batch_loss(net, rows, t, targets, np.zeros(16, np.float32)) + assert loss.item() == 0.0 + loss.backward() + assert all(torch.isfinite(p.grad).all() for p in net.parameters()) + +def test_fit_refuses_bad_weights(): + rows, t, targets = real_rows(10, seed=3) + with pytest.raises(ValueError, match="weight"): + fit_from_scratch(rows, t, targets, np.r_[np.ones(9), -1.0], seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="weight"): + fit_from_scratch(rows, t, targets, np.r_[np.ones(9), np.nan], seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="length"): + fit_from_scratch(rows, t, targets, np.ones(9), seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="1-D"): + fit_from_scratch(rows, t, targets, np.ones((10, 1)), seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="nothing to fit"): + fit_from_scratch(rows, t, targets, np.zeros(10), seed=(0,), config=SMALL) + # Finite in float64, inf in the float32 the loss computes in. + with pytest.raises(ValueError, match="finite"): + fit_from_scratch(rows, t, targets, np.r_[np.ones(9), 1e300], seed=(0,), config=SMALL) + # Finite in float32 on its own; t × weight is not, and that is the scale. + with pytest.raises(ValueError, match="fit float32"): + fit_from_scratch(rows, np.full(10, 2), targets, np.r_[np.ones(9), 2e38], seed=(0,), config=SMALL) + def test_loss_is_over_the_legal_columns_only(): # A 2-wide betting row: its loss is the sum over its two columns, so a # target met exactly at both leaves zero whatever the padding says. @@ -245,6 +330,9 @@ def test_fit_refuses_rows_or_t_that_are_not_one_dimensional_ints(): fit_from_scratch(rows[:, None], t, targets, seed=(0,), config=SMALL) with pytest.raises(ValueError, match="integer"): fit_from_scratch(rows.astype(np.float64), t, targets, seed=(0,), config=SMALL) + # A scalar: the ones default for the weight must not trip over it first. + with pytest.raises(ValueError, match="1-D"): + fit_from_scratch(np.int64(rows[0]), t[:1], targets[:1], seed=(0,), config=SMALL) # --------------------------------------------------------------------------- # The GPU path @@ -267,6 +355,7 @@ def test_the_device_loop_on_the_cpu_is_the_cpu_fit_bit_for_bit(): rows.astype(np.int64), t, targets.astype(np.float32), + np.ones(len(rows), np.float32), fit_module._fold_seed((0, 2, 1)), SMALL, device="cpu", diff --git a/tests/deepcfr/test_reservoir.py b/tests/deepcfr/test_reservoir.py index f7a89541..932d3d68 100644 --- a/tests/deepcfr/test_reservoir.py +++ b/tests/deepcfr/test_reservoir.py @@ -1,4 +1,4 @@ -"""The reservoir: Algorithm R over (row, t, target) rows, pinned on what the loop relies on. +"""The reservoir: Algorithm R over (row, t, target, weight) rows, pinned on what the loop relies on. Four claims. Below capacity every offer is kept, in order. Above it the memory never grows, counts every offer, and every offer ever made is resident with @@ -6,7 +6,8 @@ A memory saved and loaded merges the next phase exactly as the uninterrupted memory does, because the stream is a function of (seed, t, seat, memory) and the count of offers travels with the checkpoint. And the file format refuses -anything that is not a reservoir. +anything that is not a reservoir, while a file from before the weight column +existed loads with every row weighing one. """ from __future__ import annotations @@ -53,8 +54,9 @@ def filled_memory() -> Reservoir: def test_below_capacity_everything_is_kept_in_order(): memory = Reservoir(capacity=10) memory.merge(np.arange(6), targets(6), t=1, rng=np.random.default_rng(0)) - rows, t, kept = memory.contents() + rows, t, kept, weight = memory.contents() assert memory.offered == 6 and memory.size == 6 + assert weight.tolist() == [1.0] * 6 np.testing.assert_array_equal(rows, np.arange(6)) assert (t == 1).all() np.testing.assert_array_equal(kept, targets(6)) @@ -63,20 +65,23 @@ def test_a_second_merge_below_capacity_appends_after_the_first(): memory = Reservoir(capacity=10) memory.merge(np.arange(4), targets(4), t=1, rng=np.random.default_rng(0)) memory.merge(np.arange(3) + 100, targets(3), t=2, rng=np.random.default_rng(0)) - rows, t, _ = memory.contents() + rows, t, _, _ = memory.contents() assert rows.tolist() == [0, 1, 2, 3, 100, 101, 102] assert t.tolist() == [1, 1, 1, 1, 2, 2, 2] def test_never_exceeds_capacity_and_counts_every_offer(): memory = Reservoir(capacity=10) for t in range(1, 6): - memory.merge(np.arange(7) + 100 * t, targets(7), t=t, rng=np.random.default_rng(t)) + memory.merge( + np.arange(7) + 100 * t, targets(7), t=t, rng=np.random.default_rng(t), weight=np.arange(7) + 0.5 + ) assert memory.offered == 35 and memory.size == 10 - rows, t_col, kept = memory.contents() + rows, t_col, kept, weight = memory.contents() assert set(rows.tolist()) <= {r + 100 * t for t in range(1, 6) for r in range(7)} - # Each row kept its own t and its own target: the three columns move together. + # Each row kept its own t, target and weight: the four columns move together. np.testing.assert_array_equal(t_col, rows // 100) np.testing.assert_array_equal(kept[:, 0], rows % 100) + np.testing.assert_array_equal(weight, rows % 100 + 0.5) def test_a_merge_that_straddles_capacity_fills_then_samples(): memory = Reservoir(capacity=5) @@ -115,11 +120,51 @@ def test_the_later_offer_wins_a_shared_slot(): memory = Reservoir(capacity=2) memory.merge(np.array([0, 1]), targets(2), t=1, rng=np.random.default_rng(0)) memory.merge(np.array([7, 8]), targets(2), t=2, rng=FixedDraws([1, 1])) - rows, t, kept = memory.contents() + rows, t, kept, _ = memory.contents() assert rows.tolist() == [0, 8] assert t.tolist() == [1, 2] assert kept[1, 0] == 1.0 # offer 8 was the second target of its merge +def test_rows_offered_without_a_weight_weigh_one(): + _, _, _, weight = filled_memory().contents() + assert weight.dtype == np.float32 and weight.tolist() == [1.0] * 5 + +def test_a_weight_rides_its_row_through_eviction(): + # The straddle case above, with a distinct weight per offer: slot 0 is + # overwritten by offer 12, whose weight is 4.0, and the dropped offer 13 + # leaves no weight behind. + memory = Reservoir(capacity=5) + memory.merge(np.arange(3), targets(3), t=1, rng=np.random.default_rng(0), weight=np.array([0.5, 1.0, 1.5])) + memory.merge(np.arange(4) + 10, targets(4), t=2, rng=FixedDraws([0, 7]), weight=np.array([2.0, 3.0, 4.0, 5.0])) + rows, _, _, weight = memory.contents() + assert rows.tolist() == [12, 1, 2, 10, 11] + assert weight.tolist() == [4.0, 1.0, 1.5, 2.0, 3.0] + +def test_merge_refuses_bad_weights(): + memory = Reservoir(capacity=10) + rng = np.random.default_rng(0) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, -0.5])) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, np.nan])) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 1.0, 1.0])) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.ones((2, 1))) + # Finite as given, inf once cast into the float32 column: refused as the + # inf it would be stored as, not passed as the float64 it was handed as. + with pytest.raises(ValueError, match="finite"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 1e300])) + # The refit's scale is t × weight in float32: a weight the column holds + # but the product cannot is refused at the t that overflows it. + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 2e38], np.float32)) + with pytest.raises(ValueError, match="fit float32"): + memory.merge(np.arange(2), targets(2), t=2, rng=rng, weight=np.array([1.0, 2e38], np.float32)) + # A bool array is a caller handing the wrong array, not a count of samples. + with pytest.raises(ValueError, match="float or int"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([True, True])) + assert memory.offered == 2 # the one accepted offer counts; a refused one changes nothing + def test_an_empty_merge_changes_nothing(): memory = filled_memory() before = [a.copy() for a in memory.contents()] @@ -183,6 +228,9 @@ def test_the_stream_is_a_function_of_its_four_words(): def test_merge_refuses_bad_rows(): rng0 = np.random.default_rng(0) memory = Reservoir(capacity=4) + # A scalar: the ones default for the weight must not trip over it first. + with pytest.raises(ValueError, match="1-D"): + memory.merge(np.int64(3), targets(1), t=1, rng=rng0) with pytest.raises(ValueError, match="same length"): memory.merge(np.arange(3), targets(2), t=1, rng=rng0) with pytest.raises(ValueError, match="4"): @@ -204,13 +252,17 @@ def test_merge_refuses_bad_rows(): Reservoir(capacity=True) def test_contents_are_read_only_and_feed_the_fit(): - rows, t, tg = filled_memory().contents() + rows, t, tg, weight = filled_memory().contents() assert rows.dtype == np.int32 and t.dtype == np.int32 and tg.dtype == np.float32 + assert weight.dtype == np.float32 assert rows.shape == (5,) and t.shape == (5,) and tg.shape == (5, TARGET_WIDTH) + assert weight.shape == (5,) with pytest.raises(ValueError): rows[0] = 1 with pytest.raises(ValueError): tg[0, 0] = 1.0 + with pytest.raises(ValueError): + weight[0] = 2.0 # --------------------------------------------------------------------------- # The file @@ -270,6 +322,97 @@ def test_load_refuses_arrays_of_the_wrong_type(tmp_path): with pytest.raises(ValueError, match="rows must be int32"): Reservoir.load(broken) +def test_a_file_written_before_the_weight_existed_loads_with_ones(tmp_path): + # Every memory saved before PR 2 of the targeted-sampling plan has no + # `weight` key; its rows were all plain samples, so they weigh one. + old = tmp_path / "advantage-p0-t00007.npz" + np.savez( + old, + rows=np.arange(5, dtype=np.int32), + t=np.full(5, 7, np.int32), + targets=targets(5), + offered=np.int64(5), + capacity=np.int64(8), + ) + loaded = Reservoir.load(old) + rows, t, kept, weight = loaded.contents() + assert weight.dtype == np.float32 and weight.tolist() == [1.0] * 5 + loaded.merge(np.arange(3) + 50, targets(3), t=8, rng=np.random.default_rng(0), weight=np.array([2.0, 2.0, 2.0])) + assert loaded.contents()[3].tolist() == [1.0] * 5 + [2.0] * 3 + +def test_load_refuses_a_weight_of_the_wrong_length_or_type(tmp_path): + def write(name, weight): + path = tmp_path / name + np.savez( + path, + rows=np.arange(3, dtype=np.int32), + t=np.ones(3, np.int32), + targets=targets(3), + weight=weight, + offered=np.int64(3), + capacity=np.int64(8), + ) + return path + + with pytest.raises(ValueError, match="short.npz"): + Reservoir.load(write("short.npz", np.ones(2, np.float32))) + with pytest.raises(ValueError, match="weight must be float32"): + Reservoir.load(write("double.npz", np.ones(3, np.float64))) + +def test_load_refuses_a_weight_with_a_bad_value_by_name(tmp_path): + # `merge` never stores one, so a NaN, a negative, or a weight whose + # t × weight overflows float32 is a corrupt or hand-edited file, refused + # by name here rather than by `fit_from_scratch` later without it. + def write(name, weight, t=1): + path = tmp_path / name + np.savez( + path, + rows=np.arange(3, dtype=np.int32), + t=np.full(3, t, np.int32), + targets=targets(3), + weight=np.array(weight, np.float32), + offered=np.int64(3), + capacity=np.int64(8), + ) + return path + + with pytest.raises(ValueError, match="nan.npz.*finite"): + Reservoir.load(write("nan.npz", [1.0, np.nan, 1.0])) + with pytest.raises(ValueError, match="negative.npz.*non-negative"): + Reservoir.load(write("negative.npz", [1.0, -1.0, 1.0])) + with pytest.raises(ValueError, match="overflow.npz.*fit float32"): + Reservoir.load(write("overflow.npz", [1.0, 2e38, 1.0], t=2)) + assert Reservoir.load(write("fine.npz", [1.0, 2e38, 1.0], t=1)).contents()[3][1] == np.float32(2e38) + +def test_load_refuses_a_corrupt_file_without_a_weight_by_name(tmp_path): + # The ones default must not run ahead of the counts check: a negative + # `offered` in a pre-column file is refused naming the file, as before. + broken = tmp_path / "negative.npz" + np.savez( + broken, + rows=np.zeros(0, np.int32), + t=np.zeros(0, np.int32), + targets=np.zeros((0, TARGET_WIDTH), np.float32), + offered=np.int64(-1), + capacity=np.int64(8), + ) + with pytest.raises(ValueError, match="negative.npz"): + Reservoir.load(broken) + +def test_load_refuses_a_scalar_weight_by_name(tmp_path): + scalar = tmp_path / "scalar.npz" + np.savez( + scalar, + rows=np.arange(3, dtype=np.int32), + t=np.ones(3, np.int32), + targets=targets(3), + weight=np.float32(1.0), + offered=np.int64(3), + capacity=np.int64(8), + ) + with pytest.raises(ValueError, match="scalar.npz"): + Reservoir.load(scalar) + def test_a_loaded_memory_can_keep_filling_to_its_capacity(tmp_path): memory = filled_memory() memory.save(tmp_path / "m.npz") diff --git a/tests/deepcfr/test_train.py b/tests/deepcfr/test_train.py index e8c51c6d..c1d57c36 100644 --- a/tests/deepcfr/test_train.py +++ b/tests/deepcfr/test_train.py @@ -24,7 +24,7 @@ import pytest from drawmaha_solver.deepcfr.collect import Collector -from drawmaha_solver.deepcfr.reservoir import Memory, Reservoir, reservoir_stream +from drawmaha_solver.deepcfr.reservoir import TARGET_WIDTH, Memory, Reservoir, reservoir_stream import drawmaha_solver.deepcfr.train as train from drawmaha_solver.deepcfr.train import ( GRID, @@ -38,6 +38,7 @@ DeepRun, Fitted, OracleLearner, + OracleModel, current_iterate, load_run, mark_name, @@ -82,8 +83,8 @@ def __init__(self, inner: Collector) -> None: self.events: list[tuple] = [] # -- the learner -- - def fit(self, rows, t, targets, *, seed) -> Fitted: - self.events.append(("fit", seed, (rows.copy(), t.copy(), targets.copy()))) + def fit(self, rows, t, targets, weight, *, seed) -> Fitted: + self.events.append(("fit", seed, (rows.copy(), t.copy(), targets.copy(), weight.copy()))) return Fitted(("model", seed), None) def write(self, table, models) -> None: @@ -248,8 +249,8 @@ def test_fit_receives_the_memory_as_it_stands_after_the_merge(collector): step(run, spy, spy) fits = spy.of("fit") for seat in SEATS: - _, (rows, t, targets) = [fit for fit in fits if fit[0][2] == seat][-1] - for given, resident in zip((rows, t, targets), run.advantage[seat].contents(), strict=True): + _, (rows, t, targets, weight) = [fit for fit in fits if fit[0][2] == seat][-1] + for given, resident in zip((rows, t, targets, weight), run.advantage[seat].contents(), strict=True): assert given.dtype == resident.dtype np.testing.assert_array_equal(given, resident) assert rows.shape[0] == run.advantage[seat].size > 0 @@ -373,11 +374,64 @@ def test_oracle_models_round_trip(tmp_path, collector): step(run, collector, learner) learner.save(run.nets[0], tmp_path / "p0-t00001.pt") back = learner.load(tmp_path / "p0-t00001.pt") - for name in ("rows", "t", "targets"): + for name in ("rows", "t", "targets", "weight"): a, b = getattr(run.nets[0], name), getattr(back, name) assert a.dtype == b.dtype np.testing.assert_array_equal(a, b) assert run.nets[0].rows.shape[0] == run.advantage[0].size + assert run.nets[0].weight.dtype == np.float32 and (run.nets[0].weight == 1.0).all() + +def test_an_oracle_model_saved_without_a_weight_loads_with_ones(tmp_path): + old = tmp_path / "p0-t00003.pt" + with old.open("wb") as file: + np.savez(file, rows=np.arange(4, dtype=np.int32), t=np.full(4, 3, np.int32), targets=np.ones((4, TARGET_WIDTH), np.float32)) + back = OracleLearner().load(old) + assert back.weight.dtype == np.float32 and back.weight.tolist() == [1.0] * 4 + +def oracle_table(*models: OracleModel) -> np.ndarray: + table = PackedTable.whole_game() + OracleLearner().write(table, models) + return table.cumulative_regret + +def test_the_oracle_weighs_a_row_by_its_weight(): + # One real row with the sample r = (0.5, −0.25, 0, 0) at t = 3. A weight + # of 2 is the row offered twice; a weight of 0 is the row never offered. + # The equality is to the bit because these are the only samples at the + # column: 0 + r + r is 2r exactly. Beside other samples the two would + # agree only up to rounding (see the `OracleLearner` docstring). + row = np.array([first_row_at(((),))] * 2, dtype=np.int32) + t = np.array([3, 3], dtype=np.int32) + r = np.array([[0.5, -0.25, 0.0, 0.0]] * 2, dtype=np.float32) + w = lambda *values: np.array(values, dtype=np.float32) # noqa: E731 + twice = oracle_table(OracleModel(row, t, r, w(1.0, 1.0))) + weighted = oracle_table(OracleModel(row[:1], t[:1], r[:1], w(2.0))) + np.testing.assert_array_equal(twice, weighted) + assert twice.any() + once = oracle_table(OracleModel(row[:1], t[:1], r[:1], w(1.0))) + with_an_inert_row = oracle_table(OracleModel(row, t, r, w(1.0, 0.0))) + np.testing.assert_array_equal(once, with_an_inert_row) + assert not np.array_equal(once, twice) + +def test_the_oracle_keeps_each_weight_on_its_own_row_across_widths(): + # Two rows of different widths in one model — the opening spot, 2 wide + # (check or pot, no bet to fold to), and the spot after pot, pot, 3 wide + # — each with its own sample and weight. `write` flattens the samples + # row by row through `legal` and repeats each weight by its row's width; + # a weight landing on a neighbour's columns would make the table differ + # from the rows offered weight-many times. Exact as above: 0 + r + r + r + # and 3r are each one rounding of the same real number. + rows = np.array([first_row_at(((),)), first_row_at(((Action.POT, Action.POT),))], dtype=np.int32) + widths = layout().width[layout().locate_rows(rows.astype(np.int64))[0]] + assert widths.tolist() == [2, 3] + t = np.array([3, 3], dtype=np.int32) + r = np.array([[0.5, -0.25, 0.0, 0.0], [-0.75, 0.375, 0.125, 0.0]], dtype=np.float32) + weighted = oracle_table(OracleModel(rows, t, r, np.array([3.0, 2.0], np.float32))) + repeated = oracle_table( + OracleModel(np.repeat(rows, [3, 2]), np.repeat(t, [3, 2]), np.repeat(r, [3, 2], axis=0), np.ones(5, np.float32)) + ) + np.testing.assert_array_equal(weighted, repeated) + swapped = oracle_table(OracleModel(rows, t, r, np.array([2.0, 3.0], np.float32))) + assert not np.array_equal(weighted, swapped) # --------------------------------------------------------------------------- # The resume state diff --git a/tests/deepcfr/test_train_torch.py b/tests/deepcfr/test_train_torch.py index 3e0ce5e5..44f9d785 100644 --- a/tests/deepcfr/test_train_torch.py +++ b/tests/deepcfr/test_train_torch.py @@ -107,15 +107,15 @@ def test_fit_reports_a_finite_loss_and_a_run_logs_it(tmp_path): run = new_run(TINY) with Collector(seed=9, workers=0) as collector: rows = collector.collect(seat=0, t=1, walks=5) - fitted = learner.fit(rows.advantage_rows, np.ones(len(rows.advantage_rows), np.int32), rows.advantage_targets, seed=(9, 1, 0)) + ones = np.ones(len(rows.advantage_rows), np.int32) + fitted = learner.fit(rows.advantage_rows, ones, rows.advantage_targets, ones.astype(np.float32), seed=(9, 1, 0)) assert isinstance(fitted.loss, float) and math.isfinite(fitted.loss) - again = learner.fit(rows.advantage_rows, np.ones(len(rows.advantage_rows), np.int32), rows.advantage_targets, seed=(9, 1, 0)) + again = learner.fit(rows.advantage_rows, ones, rows.advantage_targets, ones.astype(np.float32), seed=(9, 1, 0)) assert again.loss == fitted.loss # the held batch is a function of the fit's seed # ... but not the fit's own first minibatch, which the net has just trained on. from drawmaha_solver.deepcfr.fit import _fold_seed, batch_loss first = torch.randint(len(rows.advantage_rows), (TINY.batch,), generator=torch.Generator().manual_seed(_fold_seed((9, 1, 0)))).numpy() - ones = np.ones(len(rows.advantage_rows), np.int32) with torch.inference_mode(): trained_on = float(batch_loss(fitted.model, rows.advantage_rows[first], ones[first], rows.advantage_targets[first]).item()) assert fitted.loss != trained_on @@ -140,23 +140,29 @@ def test_the_targets_reaching_the_fit_are_the_pot_scaled_rule_targets(monkeypatc handed = {} real = fit.fit_from_scratch - def spy(rows, t, targets, *, seed, config): - handed.update(rows=rows, t=t, targets=targets) - return real(rows, t, targets, seed=seed, config=config) + def spy(rows, t, targets, weight, *, seed, config): + handed.update(rows=rows, t=t, targets=targets, weight=weight) + return real(rows, t, targets, weight, seed=seed, config=config) monkeypatch.setattr(fit, "fit_from_scratch", spy) - NetLearner(replace(TINY, pot_power=0.5)).fit(rows_, t, targets, seed=(9, 3, 0)) + # Weights distinct per row, so a dropped or reordered column would show. + weight = np.linspace(0.5, 2.0, len(t)).astype(np.float32) + NetLearner(replace(TINY, pot_power=0.5)).fit(rows_, t, targets, weight, seed=(9, 3, 0)) np.testing.assert_array_equal(handed["targets"], pot_scaled(rows_, targets, power=0.5)) assert not np.array_equal(handed["targets"], targets) np.testing.assert_array_equal(handed["t"], t) + # The weight is a sample weight and never a divisor of the target: the + # pot scaling leaves it exactly as the memory held it. + assert handed["weight"] is weight - NetLearner(replace(TINY, pot_power=0.5, rule="dcfr")).fit(rows_, t, targets, seed=(9, 3, 0)) + NetLearner(replace(TINY, pot_power=0.5, rule="dcfr")).fit(rows_, t, targets, weight, seed=(9, 3, 0)) ruled = rule_targets(targets, t, now=3, rule="dcfr") np.testing.assert_array_equal(handed["targets"], pot_scaled(rows_, ruled, power=0.5)) np.testing.assert_array_equal(handed["t"], np.ones(len(t), np.float32)) + assert handed["weight"] is weight - NetLearner(replace(TINY, pot_power=0.0)).fit(rows_, t, targets, seed=(9, 3, 0)) + NetLearner(replace(TINY, pot_power=0.0)).fit(rows_, t, targets, weight, seed=(9, 3, 0)) assert handed["targets"] is targets # power 0 hands the memory's own array through def test_pot_scaled_regrets_write_the_same_strategy_through_the_slot_rule_and_the_ledger(): diff --git a/tests/minidrawmaha/test_boost_target.py b/tests/minidrawmaha/test_boost_target.py new file mode 100644 index 00000000..9bfabe26 --- /dev/null +++ b/tests/minidrawmaha/test_boost_target.py @@ -0,0 +1,943 @@ +"""The boost target: extra samples of a row's regret, each unbiased for the exact sum the tests first pin down. + +The module under test is rung 4's boost target. Exactly (`EXACT`), for a +table row it sums, over every physical entry of the key and every opponent +hand, the regret the walk would bank there, weighted by how often the walk +arrives; sampled, it estimates that sum from play-outs that keep k of n of +each source below the row. The claims, in +order: it imports the rules and never the grader; `sigma` is the ledger's +own σ row by row; the back-solved opponent reach equals the grader's carried +vector on every board of every public point, in the full game and in a +restricted-deal one (test a); at a seat's first decision the arrival IS the +grader's frequency, and a seat's first decisions sum to one; on a small +restricted game the regret and the arrival equal a world-by-world +expectation walked through `MiniState` (test b, exact form); on a wider +restricted game they match the mean and the visit count of thousands of +`Collector` walks, row by row within four standard errors and pooled (tests +b and c); rows of the wrong seat are refused and an unreachable row is zero; +and the cost per round-2 row is recorded (test d, a benchmark). Then the +sampled forms (e): counts that are not positive ints are refused by name; a +count at or above its fan is enumeration, bit for bit; each chance source +kept one branch at a time and averaged over its branches is the exact sum +to 1e-10, in the full game and a restricted one, which pins every n/k scale +with no noise; each source alone, and all of them together, is unbiased +against the exact sum at a row where it is live, and actually moves the +target there; the range draw is unbiased where a key's two physical entries +face wide and different ranges, before and after a throw; a row's sample depends on +(seed, t, seat, row) only; play-outs shrink the spread as 1/M; an +unreachable row is zero under any sampling; and sampling a draw or round-1 +row costs a fraction of its exact sum. + +Helpers are restated rather than imported from other test modules, as the +suite's rule has it. +""" + +from __future__ import annotations + +import ast +import os +import time +from collections.abc import Mapping +from pathlib import Path + +import numpy as np +import pytest + +from drawmaha_solver.deepcfr.collect import Collector +from drawmaha_solver.minidrawmaha import exploitability as grader +from drawmaha_solver.minidrawmaha.rules.cards import DECK, SUIT_RELABELLINGS, parse_cards, relabel +from drawmaha_solver.minidrawmaha.rules.enumeration import ( + PublicPoint, + private_keys, + public_decision_points, +) +from drawmaha_solver.minidrawmaha.rules.game import ( + Action, + InfoSet, + MiniState, + NodeKind, + canonical_picture, + random_deal, +) +from drawmaha_solver.minidrawmaha.solver.boost_target import ( + HOLES, + N_STATES, + SLOTS, + EXACT, + BoostTargets, + Sampling, + board_row, + state_index, +) +from drawmaha_solver.minidrawmaha.solver.packed_table import PackedTable + +REPO = Path(__file__).parents[2] +MODULE = REPO / "src" / "drawmaha_solver" / "minidrawmaha" / "solver" / "boost_target.py" +FULL_TARGET = os.environ.get("MINIDRAWMAHA_FULL_TARGET") == "1" + +# --------------------------------------------------------------------------- +# Helpers (restated) +# --------------------------------------------------------------------------- + +def key_on(point: PublicPoint, private) -> InfoSet: + hole, thrown, board = private + return InfoSet( + player=point.player, hole=hole, discarded=thrown, board=board, + draws=point.draws, betting=point.betting, + ) + +def shape_of(point: PublicPoint): + return private_keys(board_cards=point.board_cards, discards=point.discards) + +def random_table(seed: int) -> PackedTable: + """The whole game with N(0, 1) regrets: every row plays a different, mostly sparse σ.""" + table = PackedTable.whole_game() + table.cumulative_regret[:] = np.random.default_rng(seed).standard_normal( + table.cumulative_regret.size + ) + return table + +def positive_table(seed: int) -> PackedTable: + """The whole game with regrets in (0.05, 1.05): every action has positive probability, so every line is reachable.""" + table = PackedTable.whole_game() + table.cumulative_regret[:] = 0.05 + np.random.default_rng(seed).random(table.cumulative_regret.size) + return table + +def point_index() -> dict[tuple, int]: + """Public signature → position among the 141 points, the table's order.""" + return { + (point.board_cards, point.draws, point.betting): index + for index, point in enumerate(public_decision_points()) + } + +def grader_node_to_point(game: grader.VectorGame) -> dict[int, int]: + """The grader's own decision-node indices → table point indices, by public signature.""" + by_signature = point_index() + return { + node.index: by_signature[(node.level, node.public[0], node.public[1])] + for node in game.decisions + } + +def grader_sigma(game: grader.VectorGame, sigma) -> dict[int, np.ndarray]: + """`sigma` re-keyed by the grader's node index, as `_snapshot` would hand it to the walk.""" + to_point = grader_node_to_point(game) + return {node.index: sigma[to_point[node.index]] for node in game.decisions} + +def grader_reach(game: grader.VectorGame, sigma, *, carried: int) -> dict[int, np.ndarray]: + """The grader's forward pass, stopping at nothing: seat `carried`'s reach at every node, all boards. + + `_walk`'s chance and opponent branches verbatim (the opponent here is + `carried`); the other seat's nodes pass the reach straight down, since its + actions never enter a counterfactual reach. + """ + tables = grader_sigma(game, sigma) + out: dict[int, np.ndarray] = {} + + def forward(node, reach): + out[node.index] = reach + if node.kind is NodeKind.TERMINAL: + return + if node.kind is NodeKind.CHANCE: + child = node.child + if node.deals_board: + level = game.levels[child.level] + forward(child, node.prob * reach[level.parent] * level.ok) + else: + forward(child, node.prob * reach * game.levels[node.level].ok) + return + if node.player != carried: + for edge in node.edges: + forward(edge.child, reach) + return + keys = game.keys[(node.level, node.discards[node.player])] + valid = keys.index >= 0 + safe = np.where(valid, keys.index, 0) + table = tables[node.index] + for edge in node.edges: + pushed = np.zeros_like(reach) + for column, draw in zip(edge.actions, edge.draws, strict=True): + played = reach * np.where(valid, table[safe, column], 0.0) + if draw is None: + pushed += played + else: + grader._push(pushed, played, draw) + forward(edge.child, pushed) + + forward(game.root, game.weight[carried][None, :]) + return out + +def grader_board_row(level: int, b: int) -> int: + """My board index → the grader's batch row for the same cards (a guard against a convention drift).""" + if level == 1: + return grader.board_row((DECK[b],)) + first = b // (len(DECK) - 1) + rest = b % (len(DECK) - 1) + second = rest if rest < first else rest + 1 + return grader.board_row((DECK[first], DECK[second])) + +class RowProfile(Mapping): + """A profile over the whole game read from flat σ, float64, for the grader's functions.""" + + def __init__(self, table: PackedTable, flat: np.ndarray): + self._table, self._flat = table, flat + + def __getitem__(self, key): + _, start, width = self._table.slot_of(key) + return self._flat[start : start + width] + + def __iter__(self): + return iter(self._table) + + def __len__(self): + return len(self._table) + +# -- a restricted deal the walks can sample ------------------------------------ + +_ALLOWED: list[tuple[tuple, tuple]] = [] + +def restricted_deal(rng: np.random.Generator) -> MiniState: + """Uniform over `_ALLOWED` (seat-0 hole, seat-1 hole) pairs; module-level so the collector may pickle it.""" + return MiniState(holes=_ALLOWED[int(rng.integers(len(_ALLOWED)))]) + +def allowed_pairs(seat0, seat1) -> list[tuple[tuple, tuple]]: + return [ + (tuple(sorted(a)), tuple(sorted(b))) + for a in seat0 + for b in seat1 + if not set(a) & set(b) + ] + +# -- the world-by-world expectation ------------------------------------------ + +def world_by_world(table: PackedTable, key: InfoSet, pairs) -> tuple[np.ndarray, float]: + """E[sampled target | the walk reaches `key`] and P(reach), by enumerating every world. + + Walks `MiniState` from every allowed deal (each at 1/len(pairs)), following + every chance outcome at its probability and every action of the other + seat at its σ, and every action of the key's seat unweighted — the walk + enumerates its own actions. At the key, the exact value of each action + (`ev`: chance averaged, both seats at σ) stands in for the walk's sampled + continuation, whose expectation it is. Nothing here shares code with the + module under test: it reads σ through the ledger and the rules through + the state machine. + """ + seat = key.player + width = len(key.legal_actions()) + numerator = np.zeros(width) + denominator = 0.0 + + def ev(state: MiniState) -> float: + if state.kind is NodeKind.TERMINAL: + return state.returns()[seat] + if state.kind is NodeKind.CHANCE: + return sum(q * ev(state.apply_chance(cards)) for cards, q in state.chance_outcomes()) + sigma = table[state.infoset()].strategy() + return sum( + s * ev(state.apply(action)) + for action, s in zip(state.legal_actions(), sigma, strict=True) + if s > 0.0 + ) + + def consistent(state: MiniState) -> bool: + """Could this history still reach `key`? Its public record must be a prefix of the key's, and the seat's own cards must fit.""" + if len(state.board) > len(key.board) or len(state.draws) > len(key.draws): + return False + if state.draws != key.draws[: len(state.draws)]: + return False + for round_index, line in enumerate(state.betting): + if round_index >= len(key.betting) or key.betting[round_index][: len(line)] != line: + return False + if round_index < len(state.betting) - 1 and line != key.betting[round_index]: + return False + own = set(state.holes[seat]) | set(state.discards[seat]) + # The seat's cards so far, in the key's suit frame, must be a subset of the key's picture. + return any( + own <= set(relabel(frame, key.hole)) | set(relabel(frame, key.discarded)) + and tuple(relabel(frame, key.board))[: len(state.board)] == state.board + for frame in SUIT_RELABELLINGS + ) + + def arrive(state: MiniState, p: float) -> None: + nonlocal denominator + if state.kind is NodeKind.TERMINAL or not consistent(state): + return + if state.kind is NodeKind.CHANCE: + for cards, q in state.chance_outcomes(): + arrive(state.apply_chance(cards), p * q) + return + actions = state.legal_actions() + if state.current_player == seat: + if state.infoset() == key: + utilities = np.array([ev(state.apply(action)) for action in actions]) + sigma = table[key].strategy() + numerator[:] += p * (utilities - sigma @ utilities) + denominator += p + return # perfect recall: the key is not below itself + for action in actions: + arrive(state.apply(action), p) + return + sigma = table[state.infoset()].strategy() + for action, s in zip(actions, sigma, strict=True): + if s > 0.0: + arrive(state.apply(action), p * s) + + for holes in pairs: + arrive(MiniState(holes=holes), 1.0 / len(pairs)) + return (numerator / denominator if denominator > 0 else numerator), denominator + +def rows_at(table: PackedTable, keys) -> np.ndarray: + return np.asarray([table.row_of(key) for key in keys], dtype=np.int64) + +def assert_pooled_fair(z: np.ndarray) -> None: + """The z of many rows, pooled: mean within 4/√n of 0, mean of z² within 4√(2/n) of 1. + + The mean of z is the sharp bias test: at n rows its standard deviation + is 1/√n, so a bias of a tenth of a standard error in every row reaches + the band at n ≈ 1,600 and is caught reliably by about 3,000 rows. The + widths take the z's as independent and normal, which they are not + quite: rows on one path share walks (a positive correlation, so the + regret band is a little narrower than four true standard deviations), + visit counts over rows come from one multinomial (negative, so that + band is conservative), and a z built from a plug-in standard error over + 40-odd samples has heavier tails than normal. The seeds are fixed, so + the runs are deterministic. The arrival bands were checked at the + gated test's seed 12 for both seats and at seeds 13, 14, 15, 16, 17, + 18 and 19 for seat 1 (2026-11-05, mean z within 0.01 of zero on every + fair selection); they are a calibrated regression gate, not a + distributional guarantee. At small n the z² band is wide: 1 ± 1.79 at + n = 10; its low side bites once the half-width drops under 1, from + n = 33 (1 ± 0.98), and is 1 ± 0.84 at n = 45. + """ + n = z.size + assert abs(np.mean(z)) < 4 / np.sqrt(n), (n, np.mean(z)) + assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / n), (n, np.mean(z**2)) + +# --------------------------------------------------------------------------- +# Layering +# --------------------------------------------------------------------------- + +def test_the_module_imports_the_rules_and_never_the_grader_or_the_net(): + # The boost feeds the trainer; a bug shared with the grader would cancel + # and certify itself. The check is static, on the import block, so a + # review of the module starts where the root instructions say it should. + tree = ast.parse(MODULE.read_text()) + imported = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imported |= {alias.name for alias in node.names} + elif isinstance(node, ast.ImportFrom): + imported.add(node.module or "") + assert not any("exploitability" in name for name in imported), imported + assert not any("deepcfr" in name for name in imported), imported + assert not any(name.startswith("torch") for name in imported), imported + assert all( + name.startswith(("drawmaha_solver.minidrawmaha.rules", "drawmaha_solver.minidrawmaha.solver.packed_table")) + for name in imported + if name.startswith("drawmaha_solver") + ), imported + +# --------------------------------------------------------------------------- +# The description +# --------------------------------------------------------------------------- + +def test_build_is_one_object_per_deal_set_over_a_shared_description(): + full = BoostTargets.build() + assert BoostTargets.build() is full # the default is cached: the description takes seconds to build + assert len(full.points) == 141 + # The reach memo's bound, as the module states it: every node with one + # board card out on the path from a decision to the root, and every + # round-2 opening decision (one per round-1 line and pair of draw counts). + d = full.description + memoised = {} + for node in d.decision_of_point: + while node is not d.root: + above, _ = d.parent[node.index] + if node.level <= 1 or (above.kind is NodeKind.CHANCE and above.deals_board): + memoised[node.index] = node + node = above + level_one = [node for node in memoised.values() if node.level <= 1] + opening = [node for node in memoised.values() if node.level == 2] + assert len(level_one) == 66 and sum(node.kind is NodeKind.DECISION for node in level_one) == 29 + assert len(opening) == 20 and all(node.kind is NodeKind.DECISION for node in opening) + assert len(level_one) * 15 + len(opening) * 210 == 5_190 + # Every physical state of the frame is one of (hole, nothing thrown | a card not in the hole). + assert N_STATES == len(HOLES) * SLOTS == 455 * 16 + assert state_index(parse_cards("2c 2d 2h"), ()) == 0 # the deck is ordered by rank, then suit + assert state_index(parse_cards("2c 2d 2h"), parse_cards("3c")) == 1 + DECK.index(parse_cards("3c")[0]) + assert board_row(parse_cards("2c")) == 0 and board_row(parse_cards("2d 2c")) == 14 + # Every key of every shape has its physical entries, and the entry counts + # are the frame's: boards × states the board allows. + counts = {(1, 0): 15 * 364, (2, 0): 210 * 286, (2, 1): 210 * 286 * 10} + for shape, expected in counts.items(): + n_keys = len(private_keys(board_cards=shape[0], discards=shape[1])) + total = 0 + sizes = set() + for position in range(n_keys): + boards, states = full.entries_of(shape, position) + assert boards.shape == states.shape and boards.size >= 1 + total += boards.size + sizes.add(boards.size) + assert total == expected + # A key is a suit orbit of one picture: 6 relabellings over its stabiliser. + assert sizes <= {1, 2, 3, 6} + +def test_sigma_is_the_ledgers_strategy_row_by_row(): + full = BoostTargets.build() + table = random_table(1) + sigma = full.sigma(table) + assert len(sigma) == 141 + for p, point in enumerate(public_decision_points()): + shape = shape_of(point) + assert sigma[p].shape == (len(shape), point.width) and sigma[p].dtype == np.float64 + for position in (0, len(shape) // 3, len(shape) - 1): + key = key_on(point, shape[position]) + np.testing.assert_allclose(sigma[p][position], table[key].strategy(), rtol=1e-15, atol=0) + with pytest.raises(ValueError, match="whole game"): + full.sigma(PackedTable.listed([key_on(public_decision_points()[0], shape_of(public_decision_points()[0])[0])])) + +# --------------------------------------------------------------------------- +# (a) the back-solved reach is the grader's carried reach +# --------------------------------------------------------------------------- + +RESTRICTED = ( + [parse_cards("2c 3c 5d"), parse_cards("2d 3d 5c"), parse_cards("4h 5h 6h")], + [parse_cards("2d 4h 6h"), parse_cards("3h 4c 4d"), parse_cards("2c 2d 6c"), parse_cards("3c 5h 6d")], +) + +@pytest.mark.parametrize("holes", [None, RESTRICTED], ids=["full", "restricted"]) +@pytest.mark.parametrize("carried", [0, 1]) +def test_reach_equals_the_graders_carried_reach_on_every_board(holes, carried): + exact = BoostTargets.build(holes=holes) + game = grader.minidrawmaha_game(holes=holes) + table = random_table(2) + sigma = exact.sigma(table) + reference = grader_reach(game, sigma, carried=carried) + to_point = grader_node_to_point(game) + checked = 0 + for node in game.decisions: + p = to_point[node.index] + for b in range(reference[node.index].shape[0]): + mine = exact.reach(sigma, carried=carried, point=p, board=b) + theirs = reference[node.index][grader_board_row(node.level, b)] + np.testing.assert_allclose(mine, theirs, rtol=1e-10, atol=1e-300) + checked += bool(theirs.any()) + assert checked > 1000 # most (point, board) pairs carry reach under a random table + +# --------------------------------------------------------------------------- +# arrival at a first decision is the grader's frequency +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("seat", [0, 1]) +def test_arrival_at_a_seats_first_decision_is_the_graders_frequency_and_sums_to_one(seat): + # Before a seat has acted, own reach is the deal alone, so the grader's + # `frequency` (own × compat) at its opening points is exactly the walk's + # arrival; and every walk reaches one of them. + exact = BoostTargets.build() + game = grader.minidrawmaha_game() + table = random_table(3) + sigma = exact.sigma(table) + reach = grader_reach(game, sigma, carried=1 - seat) + to_point = grader_node_to_point(game) + opening = [ + node for node in game.decisions + if node.player == seat and node.level == 1 and len(node.public[1][0]) == seat + ] + assert len(opening) == 1 + seat + total = 0.0 + for node in opening: + p = to_point[node.index] + keys = game.keys[(1, 0)] + valid = keys.index >= 0 + _, compat = game.showdown.sums(1, reach[node.index], discards=0, shares=False) + frequency = game.weight[seat][None, :] * compat + per_key = np.zeros(keys.n_keys) + np.add.at(per_key, keys.index[valid], frequency[valid]) + rows = np.arange(len(per_key)) + table.row_of(key_on(public_decision_points()[p], shape_of(public_decision_points()[p])[0])) + arrival = exact.arrivals(sigma, seat=seat, rows=rows) + assert arrival.shape == (len(rows),) + np.testing.assert_allclose(arrival, per_key, rtol=1e-10, atol=1e-300) + total += arrival.sum() + # The full computation agrees with the arrival-only path (a round-1 + # row's value walks the whole game below it, so a handful of rows). + some = rows[:: max(1, len(rows) // 4)] + regret, with_values = exact.targets(sigma, seat=seat, rows=some, sampling=EXACT) + assert regret.shape == (len(some), 4) and np.isfinite(regret).all() + np.testing.assert_array_equal(with_values, arrival[:: max(1, len(rows) // 4)]) + assert total == pytest.approx(1.0, abs=1e-12) + +# --------------------------------------------------------------------------- +# (b) exact form: a world-by-world expectation on a small restricted game +# --------------------------------------------------------------------------- + +SMALL = ( + # Two suit relabellings of one hole for seat 0: one key, two physical entries. + [parse_cards("2c 3c 5d"), parse_cards("2d 3d 5c")], + # An opponent set that is NOT closed under that relabelling, so the two + # entries face different ranges and the sum over entries is load-bearing. + [parse_cards("2d 4h 6h"), parse_cards("3h 4c 4d"), parse_cards("2c 2h 6c")], +) + +def small_game_keys(seat: int) -> list[InfoSet]: + """One round-2 stood-pat row, one post-draw row, and (seat 1) one draw row, all reachable from SMALL.""" + points = public_decision_points() + picks = [] + hero = SMALL[seat][0] + if seat == 1: + # Seat 1's draw after seat 0 stood pat: the value below it expands over + # 37 hero continuations and 7 boards, which the exact walk can afford. + draw = next(p for p in points if p.player == 1 and p.is_draw_decision and p.draws[0].count == 0) + picks.append((draw, (hero, (), parse_cards("4c")))) + pat = next(p for p in points if p.player == seat and p.board_cards == 2 and p.discards == 0 and len(p.betting[1]) == seat) + picks.append((pat, (hero, (), parse_cards("4c 6d")))) + threw = next(p for p in points if p.player == seat and p.board_cards == 2 and p.discards == 1 and len(p.betting[1]) == seat) + kept = hero[:2] + (parse_cards("6d")[0],) + picks.append((threw, (kept, hero[2:], parse_cards("4c 3h")))) + return [key_on(point, canonical_key(private)) for point, private in picks] + +def canonical_key(private): + hole, thrown, board = private + return canonical_picture(tuple(sorted(hole)), tuple(sorted(thrown)), board) + +@pytest.mark.parametrize("seat", [0, 1]) +def test_regret_and_arrival_equal_a_world_by_world_expectation(seat): + pairs = allowed_pairs(*SMALL) + exact = BoostTargets.build(holes=SMALL) + table = positive_table(4) + sigma = exact.sigma(table) + keys = small_game_keys(seat) + rows = rows_at(table, keys) + regret, arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=EXACT) + for key, row_regret, row_arrival in zip(keys, regret, arrival, strict=True): + expected, reach = world_by_world(table, key, pairs) + assert reach > 0, str(key) + width = len(key.legal_actions()) + assert row_arrival == pytest.approx(reach, rel=1e-9, abs=0), str(key) + np.testing.assert_allclose(row_regret[:width], expected, rtol=1e-9, atol=1e-12, err_msg=str(key)) + assert not row_regret[width:].any() + # The seat-0 keys stand for two physical entries whose weights differ, + # so a representative entry would have failed above. + if seat == 0: + for key in keys: + own = [ + world_by_world(table, key, [pair for pair in pairs if pair[0] == hole])[1] + for hole in SMALL[0] + ] + assert own[0] != pytest.approx(own[1], rel=1e-3) + +# --------------------------------------------------------------------------- +# (b) and (c) sampled: thousands of walks against the exact numbers +# --------------------------------------------------------------------------- + +WIDE = ([parse_cards("2c 3c 5d")], list(HOLES)) # one hero hole for seat 0, any opponent + +def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): + # Seat 0 holds one hole, so its rows are reached often enough to test: + # 6,000 walks reach the opening row every time and the draw rows a few + # hundred times each. Per row the sample mean is within 4 s.e. of the + # exact regret and the visit count within 4 s.e. of the exact arrival; + # pooled, the mean of z² over rows sits near 1 — a bias of half a + # standard error in every row would read 1.25 and a convention bug + # (swapped reach, a dropped entry, the wrong chance factor) far above. + walks = 6_000 + _ALLOWED[:] = allowed_pairs(*WIDE) + exact = BoostTargets.build(holes=WIDE) + table = random_table(5) + sigma = exact.sigma(table) + with Collector(seed=11, workers=0, deal=restricted_deal) as collector: + collector.table.cumulative_regret[:] = table.cumulative_regret + offered = collector.collect(seat=0, t=1, walks=walks) + rows, counts = np.unique(offered.advantage_rows, return_counts=True) + enough = counts >= 40 + rows, counts = rows[enough], counts[enough] + regret, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + # Rows are kept by their EXPECTED count, 80 or more, so the observed + # cutoff of 40 sits 4.5 standard errors below every kept row's mean and + # the selection cannot bias z: a row chosen by its observed count alone + # is there because it fluctuated high. + fair = walks * arrival >= 80 + rows, counts, regret, arrival = rows[fair], counts[fair], regret[fair], arrival[fair] + assert rows.size >= 8 + widths = table.widths()[rows] + z_regret, z_visits = [], [] + for row, count, width, exact_regret, exact_arrival in zip(rows, counts, widths, regret, arrival, strict=True): + samples = offered.advantage_targets[offered.advantage_rows == row][:, :width].astype(np.float64) + mean = samples.mean(axis=0) + se = samples.std(axis=0, ddof=1) / np.sqrt(count) + live = se > 0 + z = (mean - exact_regret[:width])[live] / se[live] + assert (np.abs(z) < 4).all(), (row, mean, exact_regret[:width], se) + z_regret += z.tolist() + # Visits: binomial over walks (a row reached twice in one walk is rarer than one in a thousand). + se_visits = np.sqrt(walks * exact_arrival * (1 - exact_arrival)) + zv = (count - walks * exact_arrival) / se_visits + assert abs(zv) < 4, (row, count, walks * exact_arrival) + z_visits.append(zv) + for z in (np.asarray(z_regret), np.asarray(z_visits)): + assert z.size >= 8 + assert_pooled_fair(z) + +@pytest.mark.skipif( + not FULL_TARGET, reason="200,000 whole-game walks on 8 workers, ~2 min; set MINIDRAWMAHA_FULL_TARGET=1" +) +@pytest.mark.parametrize("seat", [0, 1]) +def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat): + walks = 200_000 + exact = BoostTargets.build() + table = random_table(6) + sigma = exact.sigma(table) + with Collector(seed=12, workers=8, deal=random_deal) as collector: + collector.table.cumulative_regret[:] = table.cumulative_regret + offered = collector.collect(seat=seat, t=1, walks=walks) + rows, counts = np.unique(offered.advantage_rows, return_counts=True) + enough = counts >= 40 + rows, counts = rows[enough], counts[enough] + arrival = exact.arrivals(sigma, seat=seat, rows=rows) + # Kept by EXPECTED count, 80 or more: the observed cutoff of 40 is then + # 4.5 standard errors below every kept row's mean, so the selection + # cannot bias z. (Selecting on the observed count alone put seat 1's + # draw rows, whose means sit near 80, at a mean z of +1.8 on two seeds: + # the rows that cleared the cutoff were the ones that fluctuated high. + # Seat 1's commonest rows, its two opening points, expect about 110.) + fair = walks * arrival >= 80 + rows, counts, arrival = rows[fair], counts[fair], arrival[fair] + assert rows.size >= 50 + z = (counts - walks * arrival) / np.sqrt(walks * arrival * (1 - arrival)) + # The sharp tests are the pooled ones (`assert_pooled_fair`). The + # per-row cap is loose on purpose: of some 3,000 rows the largest |z| + # passes 5 in a run out of a thousand, and the deal alone can do it — at + # seed 12 one seat-1 picture (3c 3d 4h with 2c on the board, six + # relabellings) was dealt 292 times against an expected 220, 4.9 + # standard errors, in a count that involves no σ and nothing of this + # module, and 221, 219, 222 times on three other seeds. + assert (np.abs(z) < 6).all(), (np.abs(z).max(), rows[np.argmax(np.abs(z))]) + assert_pooled_fair(z) + +# --------------------------------------------------------------------------- +# Refusals and the unreachable row +# --------------------------------------------------------------------------- + +def test_rows_of_the_other_seat_and_outside_the_table_are_refused(): + exact = BoostTargets.build() + table = random_table(7) + sigma = exact.sigma(table) + points = public_decision_points() + seat1 = next(i for i, point in enumerate(points) if point.player == 1) + row_of_seat_1 = table.row_of(key_on(points[seat1], shape_of(points[seat1])[0])) + with pytest.raises(ValueError, match="seat"): + exact.targets(sigma, seat=0, rows=np.array([row_of_seat_1]), sampling=EXACT) + with pytest.raises(ValueError, match="seat"): + exact.targets(sigma, seat=2, rows=np.array([0]), sampling=EXACT) + for bad in (-1, len(table)): + with pytest.raises(IndexError): + exact.targets(sigma, seat=0, rows=np.array([bad]), sampling=EXACT) + +def test_an_unreachable_row_has_zero_arrival_and_zero_regret(): + # Seat 1 never bets or raises: every seat-0 round-1 row facing a bet is + # unreachable, and the exact target says so with zeros rather than NaN. + exact = BoostTargets.build() + table = random_table(8) + points = public_decision_points() + for p, point in enumerate(points): + if point.player == 1 and not point.is_draw_decision and Action.POT in point.actions: + column = point.actions.index(Action.POT) + for position in range(len(shape_of(point))): + _, start, width = table.slot_of(key_on(point, shape_of(point)[position])) + table.cumulative_regret[start + column] = -1.0 + if not (table.cumulative_regret[start : start + width] > 0).any(): + table.cumulative_regret[start + point.actions.index(Action.CHECK_CALL)] = 1.0 + sigma = exact.sigma(table) + facing = next( + i for i, point in enumerate(points) + if point.player == 0 and point.board_cards == 1 and not point.is_draw_decision + and point.betting[0][-1:] == (Action.POT,) + ) + rows = np.arange(len(shape_of(points[facing]))) + table.row_of(key_on(points[facing], shape_of(points[facing])[0])) + regret, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + assert not arrival.any() and not regret.any() + assert np.isfinite(regret).all() + +# --------------------------------------------------------------------------- +# (e) sampling below the row: unbiased against the exact sum, seeded per row +# --------------------------------------------------------------------------- + +def only(**sampled) -> Sampling: + """A Sampling that enumerates every source but the ones named.""" + return Sampling(**{ + "opponent_range": None, "board_cards": None, "opponent_draws": None, + "hero_draws": None, "opponent_actions": None, **sampled, + }) + +def row_on(table: PackedTable, point: int, position: int = 7) -> np.ndarray: + points = public_decision_points() + return np.array([table.row_of(key_on(points[point], shape_of(points[point])[position]))]) + +def sampled(exact, sigma, seat, rows, sampling, seeds) -> np.ndarray: + """One sampled target per seed, `(len(seeds), 4)`, t equal to the seed.""" + return np.stack([ + exact.targets(sigma, seat=seat, rows=rows, sampling=sampling, seed=0, t=s)[0][0] for s in seeds + ]) + +@pytest.mark.parametrize( + "bad", + [dict(board_cards=0), dict(hero_draws=-1), dict(opponent_actions=True), dict(opponent_range=1.5), dict(playouts=0)], + ids=lambda d: next(iter(d)), +) +def test_sampling_refuses_counts_that_are_not_positive_ints(bad): + field = next(iter(bad)) + with pytest.raises(ValueError, match=f"Sampling.{field}"): + Sampling(**bad) + +def test_sampling_counts_take_any_integer_type(): + assert Sampling(board_cards=np.int64(2), playouts=np.int32(3)).board_cards == 2 + +def test_exact_with_more_than_one_playout_is_refused(): + with pytest.raises(ValueError, match="playouts must be 1"): + only(playouts=2) + +def test_a_sampled_call_names_its_seed_and_t(): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, 4) + with pytest.raises(ValueError, match="seed"): + exact.targets(sigma, seat=0, rows=rows, sampling=Sampling(), t=0) + with pytest.raises(ValueError, match="t,|non-negative int t"): + exact.targets(sigma, seat=0, rows=rows, sampling=Sampling(), seed=0) + with pytest.raises(TypeError, match="Sampling"): + exact.targets(sigma, seat=0, rows=rows, sampling=None) + +@pytest.mark.parametrize("point", [2, 0], ids=["draw row", "round-1 row"]) +def test_a_count_at_or_above_its_fan_is_enumeration_bit_for_bit(point): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, point) + full = only(board_cards=14, opponent_draws=12, hero_draws=99) + # The fans `is_exact` reads are the description's own. + assert exact.description.levels[2].children_of.shape[1] == 14 + assert all( + draw.targets.shape[-1] == 12 + for node_edges in [n.edges for n in exact.description.decision_of_point if n.edges] + for edge in node_edges for draw in edge.draws if draw is not None + ) + assert full.is_exact + with pytest.raises(ValueError, match="playouts must be 1"): + only(board_cards=14, playouts=2) + want = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + got = exact.targets(sigma, seat=0, rows=rows, sampling=full, seed=0, t=0) + assert np.array_equal(want[0], got[0]) and np.array_equal(want[1], got[1]) + +# Each source alone at a row where it is live, then all of them at once. The +# draw-row cases walk the whole draw and round 2 below them, about 0.2 s a +# call when nothing prunes, so they take fewer seeds. At these seeds the +# smallest relative bias the 4.5-s.e. gate resolves (4.5 · s.e. / |mean|, +# measured 2026-11-06 at these rows) is about 3% for the opponent's actions, +# 8% for the range and 2–8% for the opponent's replacements; for the hero's +# replacements, the board card and the mixed forms it is 13% to over 100%, +# because those sources move the target by more than its size. Their scale +# factors are pinned exactly instead, by the branch oracle below; these +# cases check that the forms compose and that each one actually samples. +UNBIASED = [ + ("opponent_range", only(opponent_range=1), 4, 0, 400), + ("opponent_range after a throw", only(opponent_range=1), 29, 0, 400), + ("opponent_actions", only(opponent_actions=1), 4, 0, 400), + ("board_cards", only(board_cards=1), 2, 0, 200), + ("hero_draws, seat 0", only(hero_draws=1), 2, 0, 60), + ("hero_draws, seat 1", only(hero_draws=1), 3, 1, 60), + ("opponent_draws", only(opponent_draws=1), 2, 0, 60), + ("the walk's sampling, whole range", Sampling(), 2, 0, 200), + ("everything at one", Sampling(opponent_range=1), 2, 0, 200), + ("two of each, two play-outs", Sampling(playouts=2, opponent_range=2, board_cards=2, opponent_draws=2, + hero_draws=2, opponent_actions=2), 2, 0, 100), +] + +@pytest.mark.parametrize(("sampling", "point", "seat", "n"), [c[1:] for c in UNBIASED], ids=[c[0] for c in UNBIASED]) +def test_each_sampled_source_is_unbiased_and_actually_samples(sampling, point, seat, n): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, point) + want, arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=EXACT) + assert arrival[0] > 0.0 + draws = sampled(exact, sigma, seat, rows, sampling, range(n)) + spread = draws.std(axis=0, ddof=1) + live = spread > 1e-9 # above float rounding, which a z would blow up + # Not vacuous: the source really varied the target at this row. + assert live.any(), "the sampled source never moved the target here" + z = (draws.mean(axis=0)[live] - want[0][live]) / (spread[live] / np.sqrt(n)) + assert (np.abs(z) < 4.5).all(), z + # A column the sample never moves is the exact one, bit for bit or near. + np.testing.assert_allclose(draws[:, ~live], np.broadcast_to(want[0][~live], draws[:, ~live].shape), atol=1e-9) + _, sampled_arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=sampling, seed=0, t=0) + assert sampled_arrival[0] == arrival[0] == exact.arrivals(sigma, seat=seat, rows=rows)[0] + +def small_draw_row(table: PackedTable, seat: int) -> np.ndarray: + """A draw row reachable from SMALL: seat 0's (both throws below it, two physical entries) or seat 1's.""" + if seat == 1: + return rows_at(table, small_game_keys(1)[:1]) + point = public_decision_points()[2] + return rows_at(table, [key_on(point, canonical_key((SMALL[0][0], (), parse_cards("4c"))))]) + +# Each chance source crosses a path below a draw row once, and the target is +# linear in it, so keeping branch j at every position, scaled n, and +# averaging over j = 0 … n − 1 is the exact sum with no noise at all: a +# scale factor off by one branch in twelve fails at 1e-10 instead of hiding +# inside a standard error. +ORACLE = [ + ("board_cards", 14, None, 0), ("hero_draws", 12, None, 0), ("opponent_draws", 12, None, 0), + ("hero_draws", 12, None, 1), + ("board_cards", 14, SMALL, 0), ("hero_draws", 12, SMALL, 0), ("opponent_draws", 12, SMALL, 0), + ("board_cards", 14, SMALL, 1), +] + +@pytest.mark.parametrize(("source", "fan", "holes", "seat"), ORACLE, + ids=[f"{c[0]}, {'small' if c[2] else 'full'}, seat {c[3]}" for c in ORACLE]) +def test_a_chance_source_kept_one_branch_at_a_time_averages_to_the_exact_sum(source, fan, holes, seat, monkeypatch): + from drawmaha_solver.minidrawmaha.solver import boost_target + exact = BoostTargets.build(holes=holes) + table = positive_table(3) + sigma = exact.sigma(table) + if holes is None: + rows = row_on(table, 2 if seat == 0 else 3) + else: + rows = small_draw_row(table, seat) + want, arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=EXACT) + assert arrival[0] > 0.0 + branch = [0] + monkeypatch.setattr( + boost_target._Draws, "subset", + lambda self, k, n, shape: None if k is None or k >= n else np.full((*shape, 1), branch[0]), + ) + total = np.zeros_like(want) + for j in range(fan): + branch[0] = j + total += exact.targets(sigma, seat=seat, rows=rows, sampling=only(**{source: 1}), seed=0, t=0)[0] + assert np.abs(want).max() > 0.0 + np.testing.assert_allclose(total / fan, want, rtol=1e-10, atol=1e-13) + +# SMALL's seat-0 hole and its clubs ↔ diamonds relabelling against every +# hole holding more clubs than diamonds: a set the relabelling maps onto its +# complement, so the key's two physical entries face wide and different +# ranges (SMALL's three opponent holes leave one live hand at these rows, +# and a range draw there has nothing to vary). +LOPSIDED = ( + SMALL[0], + [hole for hole in HOLES if sum(c.suit == 0 for c in hole) > sum(c.suit == 1 for c in hole)], # suit 0 = c, 1 = d +) + +def test_the_range_draw_is_unbiased_where_two_entries_face_different_ranges(): + # At a stood-pat row and at one after a throw (where the thrown card + # enters the clash mask): a draw that mixed one entry's total or mask + # into the other's would shift the mean. + exact = BoostTargets.build(holes=LOPSIDED) + table = positive_table(4) + sigma = exact.sigma(table) + rows = rows_at(table, small_game_keys(0)) + want, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + assert (arrival > 0.0).all() + n = 600 + draws = np.stack([ + exact.targets(sigma, seat=0, rows=rows, sampling=only(opponent_range=1), seed=0, t=s)[0] for s in range(n) + ]) # (n, rows, 4) + spread = draws.std(axis=0, ddof=1) + live = spread > 1e-9 + assert live[0].any() and live[1].any(), spread + z = (draws.mean(axis=0)[live] - want[live]) / (spread[live] / np.sqrt(n)) + assert (np.abs(z) < 4.5).all(), z + +def test_a_rows_sample_depends_on_seed_t_seat_and_row_only(): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = np.concatenate([row_on(table, 4, 7), row_on(table, 2, 11), row_on(table, 4, 30)]) + at = lambda r, t=5: exact.targets(sigma, seat=0, rows=r, sampling=Sampling(opponent_range=1), seed=1, t=t)[0] + together = at(rows) + assert np.array_equal(at(rows[::-1]), together[::-1]) + assert np.array_equal(np.concatenate([at(rows[:1]), at(rows[1:])]), together) + assert np.array_equal(at(rows[[0, 0, 1, 2]]), together[[0, 0, 1, 2]]) + assert not np.array_equal(at(rows, t=6), together) + +def test_more_playouts_shrink_the_spread_as_one_over_m(): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, 4) + one = sampled(exact, sigma, 0, rows, only(opponent_actions=1), range(200)).var(axis=0, ddof=1) + eight = sampled(exact, sigma, 0, rows, only(opponent_actions=1, playouts=8), range(200)).var(axis=0, ddof=1) + live = one > 0.0 + ratio = eight[live] / one[live] + assert ((1 / 16 < ratio) & (ratio < 1 / 4)).all(), ratio + +def test_an_unreachable_row_is_zero_under_any_sampling(): + # The same construction as the exact test above: seat 1 never bets, so a + # seat-0 round-1 row facing a bet has no weight to draw a range from. + exact = BoostTargets.build() + table = random_table(8) + points = public_decision_points() + for p, point in enumerate(points): + if point.player == 1 and not point.is_draw_decision and Action.POT in point.actions: + column = point.actions.index(Action.POT) + for position in range(len(shape_of(point))): + _, start, width = table.slot_of(key_on(point, shape_of(point)[position])) + table.cumulative_regret[start + column] = -1.0 + if not (table.cumulative_regret[start : start + width] > 0).any(): + table.cumulative_regret[start + point.actions.index(Action.CHECK_CALL)] = 1.0 + sigma = exact.sigma(table) + facing = next( + i for i, point in enumerate(points) + if point.player == 0 and point.board_cards == 1 and not point.is_draw_decision + and point.betting[0][-1:] == (Action.POT,) + ) + rows = np.arange(4) + table.row_of(key_on(points[facing], shape_of(points[facing])[0])) + regret, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=Sampling(opponent_range=1), seed=0, t=0) + assert not arrival.any() and not regret.any() + assert np.isfinite(regret).all() + +# --------------------------------------------------------------------------- +# (d) cost per row: a benchmark, recorded, with a loose ceiling +# --------------------------------------------------------------------------- + +def test_a_round_two_row_costs_a_few_milliseconds(): + exact = BoostTargets.build() + table = random_table(9) + sigma = exact.sigma(table) + rng = np.random.default_rng(0) + points = public_decision_points() + offsets = np.cumsum([0] + [len(shape_of(point)) for point in points]) + round_two = [p for p, point in enumerate(points) if point.player == 0 and point.board_cards == 2] + rows = np.concatenate([ + offsets[p] + rng.choice(len(shape_of(points[p])), size=10, replace=False) for p in round_two + ]) + exact.targets(sigma, seat=0, rows=rows[:5], sampling=EXACT) # warm the description + started = time.perf_counter() + exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + per_row = (time.perf_counter() - started) / rows.size + print(f"\nexact target: {per_row * 1e3:.2f} ms per round-2 row over {rows.size} rows") + # The plan's figure is 1 ms a row and the measured one 0.8 ms. The + # ceiling is fifty times that: wide enough that a laptop busy with a + # training run (0.95 ms measured beside a 9-worker LCFR run) cannot + # cross it, narrow enough to catch a regression to the grader's + # whole-board update, which is about a second a row. + assert per_row < 0.050 + +def test_sampling_below_a_draw_or_round_one_row_costs_a_fraction_of_the_exact_sum(): + # The board card's fourteen branches and the hero's thirty-six + # replacements are what an exact draw or round-1 row pays for; the walk's + # sampling keeps one of each. Printed for the record; the assertion is a + # ratio, which a busy laptop slows on both sides alike. + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + lines = [] + for point, name in ((4, "round 2"), (2, "draw"), (0, "round 1")): + rows = np.concatenate([row_on(table, point, position) for position in (3, 7, 11)]) + exact.targets(sigma, seat=0, rows=rows[:1], sampling=EXACT) # warm + cost = {} + for label, sampling in (("exact", EXACT), ("walk's sampling", Sampling())): + started = time.perf_counter() + exact.targets(sigma, seat=0, rows=rows, sampling=sampling, seed=0, t=0) + cost[label] = (time.perf_counter() - started) / rows.size + lines.append(f"{name:>8}: exact {cost['exact'] * 1e3:7.1f} ms, walk's sampling {cost["walk's sampling"] * 1e3:6.1f} ms a row") + if point != 4: + assert cost["walk's sampling"] < cost["exact"] / 2, (name, cost) + print("\nboost target, a positive table:\n" + "\n".join(lines))