From 9c817345436fed806bcb5e95238c735c5649cdd3 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 14:21:02 -0500 Subject: [PATCH 01/16] =?UTF-8?q?feat(rung4):=20the=20exact=20target=20?= =?UTF-8?q?=E2=80=94=20a=20row's=20sampled=20regret=20averaged=20over=20ev?= =?UTF-8?q?ery=20opponent=20hand=20and=20every=20physical=20entry=20of=20i?= =?UTF-8?q?ts=20key,=20and=20the=20walk's=20arrival=20rate,=20from=20the?= =?UTF-8?q?=20rules=20alone?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR 1 of the targeted-sampling plan. `minidrawmaha/solver/exact_target.py` computes, for any table row, the regret the walk's samples there estimate (the mean over the opponent's hands of u − ⟨σ, u⟩, weighted by the deck and the opponent's own play, normalised by the arrival) and the probability one walk visits the row. It re-derives the public tree, key maps, draw maps and showdown from rules/ and never imports the grader, which grades what this will feed. Measured: a round-2 row costs 0.8 ms, the arrival alone 0.3 ms; a draw row 45–85 ms and a round-1 row 0.2–0.6 s, because the value below expands over the traverser's own replacements and the second board card. Tests: the back-solved reach equals the grader's carried vector on every board of every point (1e-10, full and restricted deals); arrival at a seat's first decision is the grader's frequency and sums to one; regret and arrival equal a world-by-world walk through MiniState on a small restricted game (1e-9) and the mean and visit count of 6,000 Collector walks row by row within 4 s.e. and pooled; the module's import block holds no grader. --- src/drawmaha_solver/minidrawmaha/AGENTS.md | 6 +- .../minidrawmaha/solver/exact_target.py | 872 ++++++++++++++++++ tests/AGENTS.md | 3 +- tests/minidrawmaha/test_exact_target.py | 622 +++++++++++++ 4 files changed, 1501 insertions(+), 2 deletions(-) create mode 100644 src/drawmaha_solver/minidrawmaha/solver/exact_target.py create mode 100644 tests/minidrawmaha/test_exact_target.py diff --git a/src/drawmaha_solver/minidrawmaha/AGENTS.md b/src/drawmaha_solver/minidrawmaha/AGENTS.md index 56f5ded6..4dbc0ec7 100644 --- a/src/drawmaha_solver/minidrawmaha/AGENTS.md +++ b/src/drawmaha_solver/minidrawmaha/AGENTS.md @@ -13,7 +13,11 @@ Three subpackages, and one module that stays alone on purpose: arrays), `infoset_table` (a ledger window over it), `mccfr` (external sampling, one hand per seat), `lockstep` (W hands per seat against one frozen σ; the reference), `parallel` (the same in W processes over shared - memory), `lockstep_run` (resume, grid checkpoints, progress). + memory), `lockstep_run` (resume, grid checkpoints, progress), and + `exact_target` (rung 4's boost oracle: the regret a row's sampled targets + estimate, summed over every opponent hand and every physical entry of the + key, and the probability a walk arrives there; rules-only, and it must + never import `exploitability.py`, which grades what it feeds). - `answer/` — everything built on the solved game. `strategy` (the frozen LCFR average, a release asset pinned by SHA-256), `analysis` (`minidraw-analysis`, writes `figures/rung3/`), `play` (`minidraw-play`), diff --git a/src/drawmaha_solver/minidrawmaha/solver/exact_target.py b/src/drawmaha_solver/minidrawmaha/solver/exact_target.py new file mode 100644 index 00000000..66b5fcb6 --- /dev/null +++ b/src/drawmaha_solver/minidrawmaha/solver/exact_target.py @@ -0,0 +1,872 @@ +"""The exact target: what the walk's regret sample at a table row is worth on average, computed. + +Rung 4 learns every infoset from the regret samples the walks bank there, +`r = u − ⟨σ, u⟩` with one opponent hand and one line of opponent play drawn +per visit (`mccfr.traverse`, `lockstep.walk`). At a rare round-2 row a visit +comes once in thousands of iterations and its sample swings by chips, almost +all of it the opponent's hidden hand. This module computes, for any table +row, the quantity those samples estimate — the regret averaged over every +hand the opponent could hold, weighted by how often the deck and the +opponent's own play bring exactly that hand there — and the probability that +one walk arrives at the row at all. A later stage (the "boost") adds one row +with that regret at the weight the walk's samples would have carried, so a +targeted infoset is learned from an exact number instead of a handful of +noisy ones. + +**What is computed.** For a row standing for the key I of seat `seat`: + + regret(I, a) = Σ_{b,h,o} w(b,h,o) · (v(a | b,h,o) − v(I | b,h,o)) / Σ_{b,h,o} w(b,h,o) + arrival(I) = Σ_{b,h,o} w(b,h,o) + w(b,h,o) = deal(h) · r_b[o] · compat_{b,h}[o] + +where (b, h) runs over every PHYSICAL entry of the key — a board row in the +order it came, and the seat's physical (hole, thrown card) — and o over the +opponent's physical states. `r_b[o]` is the reach of the deck and the +opponent at the row's public point on board b: the opponent's deal weight, +the chance probability of every card dealt along the path, and the +opponent's σ at each of its decisions, pushed through its draw. `compat` +is one where the two seats' cards are disjoint. `deal(h)` is the +traverser's own deal probability — seat 0 carries 1/100,100 per allowed +pair and seat 1 carries 1 (the grader's convention in `_weights`, restated +here) — and at a post-draw state it is the sum over the three pre-draw holes +that could have become it, because the key forgets which card was drawn and +the walk arrives through any of them. `v(a | b,h,o)` is the traverser's +expected chips below action a with both seats at σ and the deck averaged. + +**Why the ratio, and why the traverser's own σ never enters the weight.** +The walk enumerates the traverser's own actions, so how often it arrives at +I is the deck's and the opponent's doing alone: `arrival` is exactly the +expected number of visits per walk, and `regret` is the conditional mean of +the banked sample given a visit — `E[r | reach]`. Normalising by Σ w is what +makes the exact row commensurable with the sampled ones: a sampled row's +target is one draw of r, so its expectation is this ratio, and the boost's +row weight, `K · arrival · t`, restores the total weight the K walks would +have put there. Dropping the denominator, or weighting every target 1, is +the biased "teleport" the plan rules out. The traverser's own σ enters +only inside `v(I) = ⟨σ, v(·)⟩`, the regret's baseline, as it does in the +walk. + +**Why every physical entry is summed.** A key is a suit orbit of one +picture — up to six physical (board, state) pairs, fewer when the picture +has a suit symmetry. In the full game a global suit relabelling maps the +whole game onto itself and σ is read in key space, so the six entries +carry identical numbers and one representative times the orbit size would +do. The module sums them anyway: a restricted deal set (`build(holes=)`, +which PR 4's oracle test and this module's own tests use) is not closed +under relabelling, and there the entries face different opponent ranges +and the sum is load-bearing. The orbit shortcut is an optimisation left for +the day the full game's cost matters, with the sum as its oracle. + +**Card removal, applied once and in the same place.** The numerator's +weights and the denominator's are the same `compat` vector, computed by +`_compat` from the two seats' card sets. Below the row nothing further is +masked by hand: a replacement the deck could not deal — the board card, a +card the other seat holds — leads to a terminal where the two seats share a +card and pays zero through the same compat, and the chance node's public +1/|stub| is then exactly the probability of each live replacement, because +the stub's size is public (the walk relies on the same fact). The board-2 +deal masks the opponent's states the same way the grader's levels do. + +**The frame.** Every vector is indexed by a seat's physical state, 455 +holes in deck order × 16 slots (slot 0 for nothing thrown, 1 + c for card +c), 7,280 entries, with the board as a batch axis: 15 rows after the first +card, 210 ORDERED pairs after the second. Canonical keys enter at one place, +a map from (board row, state) to a position in `private_keys(shape)`, used +to gather σ rows; the same map inverted lists each key's physical entries. +That is the grader's frame, re-derived here from `rules/` alone. + +**Rules-only, on purpose.** This module imports `rules/` and the packed +table and nothing from `exploitability.py`, which grades what the boost +feeds; a tree, a key map or a showdown table shared between the two would +let one bug certify itself. The tests hold the two descriptions to each +other instead: the back-solved reach equals the grader's carried vector on +every board of every point to 1e-10, and the regret and arrival equal a +world-by-world walk through `MiniState` and the mean of thousands of real +walks. + +**Cost, measured on this Mac (2026-11-05, a random table).** The description +is built once per process, 5–7 s — the key maps, the draw maps, the +public tree with parent links, the showdown ranks — and holds no showdown +matrix: shares and compatibility are computed per terminal from the holes' +ranks and card sets, folded over the opponent's 455 holes. `sigma` over the +whole table takes 0.12 s. A round-2 row costs 0.8 ms (1.5 ms when 2,240 rows +are spread over every round-2 point, where the reach memo warms less per +row), and the arrival alone 0.3 ms. A draw row costs 45–85 ms and a round-1 +row 0.2–0.6 s, because the value below them expands over the traverser's own +replacements (36 states per throw) and the second board card (14 boards): +the opponent's vector is carried once per board and shared by every hero +state on it, which is what keeps those within a second. The opponent's reach +is memoised per call at every level-1 point (29 of them, 15 boards each) and +at every round-2 opening point (20: one per round-1 line and pair of draw +counts, 210 boards each), 58 KB a (point, board) pair, so the memo is at +most 29 × 15 + 20 × 210 = 4,635 vectors, about 270 MB; a call over 2,240 +round-2 rows peaked at about 260 MB. `tests/minidrawmaha/test_exact_target.py` prints the +round-2 figure on every run. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from functools import cache +from itertools import combinations, permutations +from typing import Any + +import numpy as np + +from drawmaha_solver.minidrawmaha.rules.cards import DECK, Card +from drawmaha_solver.minidrawmaha.rules.enumeration import ( + PublicPoint, + private_keys, + public_decision_points, +) +from drawmaha_solver.minidrawmaha.rules.game import ( + BETTING_ACTIONS, + DRAW_ACTIONS, + Action, + MiniState, + NodeKind, + canonical_picture, + chip_state, + draw_order, + throw_count, +) +from drawmaha_solver.minidrawmaha.rules.hands import HOLE_CARDS, inner_score, outer_score +from drawmaha_solver.minidrawmaha.solver.packed_table import PackedTable + +Hole = tuple[Card, ...] +SEATS = (0, 1) + +# The widest ledger: the draw's four throws. Regret rows are padded to it. +TARGET_WIDTH = max(len(DRAW_ACTIONS), len(BETTING_ACTIONS)) + +# --------------------------------------------------------------------------- +# The frame: physical states and board rows +# --------------------------------------------------------------------------- + +HOLES: tuple[Hole, ...] = tuple(combinations(DECK, HOLE_CARDS)) +SLOTS = 1 + len(DECK) +N_STATES = len(HOLES) * SLOTS + +_HOLE_INDEX = {hole: index for index, hole in enumerate(HOLES)} +_BOARD_SETS = tuple(combinations(DECK, 2)) +_SET_INDEX = {pair: index for index, pair in enumerate(_BOARD_SETS)} +_LEVEL_ROWS = (1, len(DECK), len(DECK) * (len(DECK) - 1)) + +def state_index(hole: Hole, thrown: Hole) -> int: + """The row of `(hole, thrown)` in a seat's vector; `thrown` is empty or one card.""" + slot = 0 if not thrown else 1 + DECK.index(thrown[0]) + return _HOLE_INDEX[tuple(sorted(hole))] * SLOTS + slot + +def board_row(board: tuple[Card, ...]) -> int: + """A board's batch row: deck order for one card, `(first, second)` ordered for two. + + Ordered pairs, 210 rows rather than 105 sets, because round 1 was played + against the first card alone and the two orders carry different reach + into round 2; only the showdown, which cannot tell them apart, is per set. + """ + if len(board) == 1: + return DECK.index(board[0]) + first, second = DECK.index(board[0]), DECK.index(board[1]) + return first * (len(DECK) - 1) + (second if second < first else second - 1) + +def _state_cards() -> tuple[np.ndarray, np.ndarray]: + """Per state: which deck cards it accounts for (hole and thrown), and whether it is a real state.""" + holds = np.zeros((N_STATES, len(DECK)), dtype=bool) + valid = np.zeros(N_STATES, dtype=bool) + for h, hole in enumerate(HOLES): + for slot in range(SLOTS): + s = h * SLOTS + slot + for card in hole: + holds[s, DECK.index(card)] = True + if slot == 0: + valid[s] = True + elif not holds[s, slot - 1]: + holds[s, slot - 1] = True + valid[s] = True + return holds, valid + +# --------------------------------------------------------------------------- +# The description: levels, key maps, draw maps, the public tree, the showdown ranks +# --------------------------------------------------------------------------- + +@dataclass(frozen=True, slots=True, eq=False) +class _Level: + """The boards at one board count: `ok[b, s]` is 1 when state s avoids board b; `parent[b]` is the board above b and `children_of[p]` the boards of this level under p.""" + + ok: np.ndarray + parent: np.ndarray + children_of: np.ndarray + +@dataclass(frozen=True, slots=True, eq=False) +class _KeyMap: + """(board row, state) → position in one shape's `private_keys`, −1 outside its support; and the inverse.""" + + index: np.ndarray # int32 (n_boards, N_STATES) + n_keys: int + entry_start: np.ndarray # int64 (n_keys + 1,): entries of key k are entry_*[start[k]:start[k+1]] + entry_board: np.ndarray # int32 + entry_state: np.ndarray # int32 + +@dataclass(frozen=True, slots=True, eq=False) +class _DrawMap: + """One throw as a map from pre-draw holes to post-draw states: `targets[b, k, j]` on board b, hole k, replacement j.""" + + source: np.ndarray # int32 (455,): the pre-draw state of each hole + targets: np.ndarray # int32 (15, 455, 12) + +@dataclass(frozen=True, slots=True, eq=False) +class _Edge: + """The actions of a decision that lead to one child, with the draw map of each throw (None for a bet or standing pat).""" + + child: _Node + actions: tuple[int, ...] + draws: tuple[_DrawMap | None, ...] + +@dataclass(frozen=True, slots=True, eq=False) +class _Node: + """One public node. `point` is a decision's index among the 141 table points; `deals_board` tells a board deal from a replacement.""" + + index: int + kind: NodeKind + level: int + discards: tuple[int, int] + player: int | None + actions: tuple[Any, ...] + point: int | None + edges: tuple[_Edge, ...] + child: _Node | None + prob: float + deals_board: bool + fold_winner: int | None + stake: float + pot: float + committed: tuple[float, float] + +@dataclass(frozen=True, slots=True, eq=False) +class _Description: + """Everything that does not depend on the deal set, built once per process.""" + + points: tuple[PublicPoint, ...] + row_offset: np.ndarray + column_offset: np.ndarray + width: np.ndarray + holds: np.ndarray # bool (N_STATES, 15) + valid: np.ndarray # bool (N_STATES,) + hole_of: np.ndarray # int32 (N_STATES,) + hole_holds: np.ndarray # float64 (455, 15): which cards each hole holds + levels: tuple[_Level, _Level, _Level] + set_of_row: np.ndarray # int32 (210,): level-2 row → board set, for the showdown alone + keys: dict[tuple[int, int], _KeyMap] + root: _Node + parent: dict[int, tuple[_Node, _Edge | None]] + decision_of_point: tuple[_Node, ...] + inner: np.ndarray # int32 (455,): dense rank of each hole's inner hand + outer: np.ndarray # int32 (105, 455): dense rank of each hole's outer hand with each board set, 0 where they clash + +def _levels(holds: np.ndarray, valid: np.ndarray) -> tuple[_Level, _Level, _Level]: + n = len(DECK) + # `parent` and `children_of` both live on the CHILD level: `parent[b]` is + # the board above b, `children_of[p]` every board of this level under p. + zero = _Level( + ok=valid[None, :].astype(np.float64), + parent=np.zeros(1, np.int32), + children_of=np.zeros((0, 0), np.int32), + ) + one = _Level( + ok=(valid[None, :] & ~holds.T).astype(np.float64), + parent=np.zeros(n, np.int32), + children_of=np.arange(n, dtype=np.int32)[None, :], + ) + ok = np.zeros((n * (n - 1), N_STATES)) + parent = np.zeros(n * (n - 1), np.int32) + for first in range(n): + for second in range(n): + if first != second: + row = board_row((DECK[first], DECK[second])) + ok[row] = valid & ~holds[:, first] & ~holds[:, second] + parent[row] = first + two = _Level( + ok=ok, + parent=parent, + children_of=np.stack([np.flatnonzero(parent == first) for first in range(n)]).astype(np.int32), + ) + return zero, one, two + +def _key_map(level: int, discards: int) -> _KeyMap: + """(board row, state) → position in `private_keys(level, discards)`, built per ORDERED board row, and its inverse.""" + keys = private_keys(board_cards=level, discards=discards) + position = {key: k for k, key in enumerate(keys)} + boards = [(card,) for card in DECK] if level == 1 else list(permutations(DECK, 2)) + index = np.full((_LEVEL_ROWS[level], N_STATES), -1, np.int32) + for board in boards: + b = board_row(board) + for h, hole in enumerate(HOLES): + if set(hole) & set(board): + continue + if discards == 0: + index[b, h * SLOTS] = position[canonical_picture(hole, (), board)] + continue + for card in DECK: + if card in hole or card in board: + continue + index[b, h * SLOTS + 1 + DECK.index(card)] = position[ + canonical_picture(hole, (card,), board) + ] + flat = index.ravel() + order = np.argsort(flat, kind="stable") + order = order[flat[order] >= 0] + sorted_keys = flat[order] + start = np.searchsorted(sorted_keys, np.arange(len(keys) + 1)).astype(np.int64) + return _KeyMap( + index=index, + n_keys=len(keys), + entry_start=start, + entry_board=(order // N_STATES).astype(np.int32), + entry_state=(order % N_STATES).astype(np.int32), + ) + +def _draw_maps() -> tuple[_DrawMap, _DrawMap, _DrawMap]: + """The three one-card throws as maps, one board card out, every replacement listed. + + Which physical card `THROW_LOW` names depends on the board through the + canonical relabelling, so the thrown card is read off `draw_order`. + """ + n = len(DECK) + replacements = n - HOLE_CARDS + targets = np.zeros((n, len(HOLES), HOLE_CARDS, replacements), np.int32) + for b, board_card in enumerate(DECK): + for h, hole in enumerate(HOLES): + ordered = draw_order(hole=hole, discarded=(), board=(board_card,)) + for k, thrown in enumerate(ordered): + kept = tuple(card for card in hole if card != thrown) + spare = [card for card in DECK if card not in hole] + for j, drawn in enumerate(spare): + targets[b, h, k, j] = state_index(kept + (drawn,), (thrown,)) + source = (np.arange(len(HOLES)) * SLOTS).astype(np.int32) + return tuple(_DrawMap(source=source, targets=targets[:, :, k, :]) for k in range(HOLE_CARDS)) + +def _public_tree( + draws: tuple[_DrawMap, ...], points: tuple[PublicPoint, ...] +) -> tuple[_Node, dict[int, tuple[_Node, _Edge | None]], tuple[_Node, ...]]: + """The public tree with edges and parent links, walked from one representative deal. + + Replacement deals are told apart from board deals (the census merges + them), and a decision is tied to its table point by public signature. + """ + by_signature = { + (point.board_cards, point.draws, point.betting): index for index, point in enumerate(points) + } + root_state = MiniState(holes=(DECK[0:HOLE_CARDS], DECK[HOLE_CARDS : 2 * HOLE_CARDS])) + nodes: dict[tuple, _Node] = {} + order: list[_Node | None] = [] + parent: dict[int, tuple[_Node, _Edge | None]] = {} + decision_of_point: dict[int, _Node] = {} + + def visit(state: MiniState) -> _Node: + signature = ( + state.kind, + len(state.board), + state.draws, + state.betting, + tuple(len(hole) for hole in state.holes), + ) + if signature in nodes: + return nodes[signature] + index = len(order) + order.append(None) + fields: dict[str, Any] = dict( + index=index, + kind=state.kind, + level=len(state.board), + discards=tuple(len(pile) for pile in state.discards), + player=None, + actions=(), + point=None, + edges=(), + child=None, + prob=0.0, + deals_board=False, + fold_winner=None, + stake=0.0, + pot=0.0, + committed=(0.0, 0.0), + ) + if state.kind is NodeKind.TERMINAL: + chips = chip_state(state.betting) + line = state.betting[-1] + if line and line[-1] is Action.FOLD: + loser = (len(line) - 1) % 2 + fields.update(fold_winner=1 - loser, stake=float(chips.committed[loser])) + else: + fields.update( + pot=float(chips.pot), + committed=(float(chips.committed[0]), float(chips.committed[1])), + ) + node = _Node(**fields) + elif state.kind is NodeKind.CHANCE: + outcomes = state.chance_outcomes() + child = visit(state.apply_chance(outcomes[0][0])) + fields.update( + child=child, + prob=outcomes[0][1], + # A short hole means the deck owes a replacement; otherwise a board card. + deals_board=all(len(hole) == HOLE_CARDS for hole in state.holes), + ) + node = _Node(**fields) + parent[child.index] = (node, None) + else: + actions = state.legal_actions() + grouped: dict[int, tuple[list[int], list[_DrawMap | None]]] = {} + children: dict[int, _Node] = {} + for k, action in enumerate(actions): + child = visit(state.apply(action)) + children[child.index] = child + columns, maps = grouped.setdefault(child.index, ([], [])) + columns.append(k) + maps.append( + draws[DRAW_ACTIONS.index(action) - 1] + if state.is_draw_decision() and throw_count(action) > 0 + else None + ) + edges = tuple( + _Edge(child=children[child_index], actions=tuple(columns), draws=tuple(maps)) + for child_index, (columns, maps) in grouped.items() + ) + fields.update( + player=state.current_player, + actions=actions, + point=by_signature[(len(state.board), state.draws, state.betting)], + edges=edges, + ) + node = _Node(**fields) + for edge in edges: + parent[edge.child.index] = (node, edge) + decision_of_point[node.point] = node + nodes[signature] = node + order[index] = node + return node + + root = visit(root_state) + if len(decision_of_point) != len(points): + raise AssertionError("the public tree and the enumerator disagree on the decision points") + return root, parent, tuple(decision_of_point[p] for p in range(len(points))) + +def _dense_rank(scores: Sequence[Any]) -> np.ndarray: + """Scores as integers with the same order, so a sign of a difference is a comparison.""" + ladder = {score: rank for rank, score in enumerate(sorted(set(scores)))} + return np.array([ladder[score] for score in scores], dtype=np.int32) + +def _showdown_ranks() -> tuple[np.ndarray, np.ndarray]: + """Per hole, the dense rank of its inner hand, and of its outer hand with each board set (0 where the hole clashes).""" + inner = _dense_rank([inner_score(hole) for hole in HOLES]) + outer = np.zeros((len(_BOARD_SETS), len(HOLES)), np.int32) + for s, board in enumerate(_BOARD_SETS): + live = [h for h, hole in enumerate(HOLES) if not set(hole) & set(board)] + outer[s, live] = _dense_rank([outer_score(HOLES[h], board) for h in live]) + return inner, outer + +@cache +def _description() -> _Description: + points = public_decision_points() + sizes = [len(private_keys(board_cards=p.board_cards, discards=p.discards)) for p in points] + widths = [p.width for p in points] + holds, valid = _state_cards() + hole_of = (np.arange(N_STATES) // SLOTS).astype(np.int32) + set_of_row = np.zeros(_LEVEL_ROWS[2], np.int32) + for first in DECK: + for second in DECK: + if first != second: + set_of_row[board_row((first, second))] = _SET_INDEX[tuple(sorted((first, second)))] + root, parent, decision_of_point = _public_tree(_draw_maps(), points) + inner, outer = _showdown_ranks() + return _Description( + points=points, + row_offset=np.concatenate([[0], np.cumsum(sizes)]).astype(np.int64), + column_offset=np.concatenate( + [[0], np.cumsum([s * w for s, w in zip(sizes, widths, strict=True)])] + ).astype(np.int64), + width=np.asarray(widths, dtype=np.int64), + holds=holds, + valid=valid, + hole_of=hole_of, + hole_holds=holds[::SLOTS].astype(np.float64), + levels=_levels(holds, valid), + set_of_row=set_of_row, + keys={shape: _key_map(*shape) for shape in ((1, 0), (2, 0), (2, 1))}, + root=root, + parent=parent, + decision_of_point=decision_of_point, + inner=inner, + outer=outer, + ) + +# --------------------------------------------------------------------------- +# The deal set +# --------------------------------------------------------------------------- + +def _weights(holes: tuple[Iterable[Hole], Iterable[Hole]] | None) -> tuple[np.ndarray, np.ndarray]: + """Per-seat deal probability at pre-draw states, zero elsewhere: uniform over the allowed disjoint pairs. + + P(r, h) = weight[0][r] · weight[1][h] · [r ∩ h = ∅]; 1/100,100 in the + full game, carried by seat 0 with seat 1 at 1 — the walk's `random_deal` + distribution, and the grader's convention. + """ + allowed = ( + (set(HOLES), set(HOLES)) + if holes is None + else (set(map(tuple, map(sorted, holes[0]))), set(map(tuple, map(sorted, holes[1])))) + ) + for seat, chosen in enumerate(allowed): + unknown = chosen - set(HOLES) + if unknown: + raise ValueError(f"seat {seat} cannot be dealt {sorted(unknown)}") + pairs = sum(1 for r in allowed[0] for h in allowed[1] if not set(r) & set(h)) + if pairs == 0: + raise ValueError("no deal gives both seats a hole from their allowed sets") + out = [] + for seat, chosen in enumerate(allowed): + weight = np.zeros(N_STATES) + weight[::SLOTS] = [1.0 if hole in chosen else 0.0 for hole in HOLES] + if seat == 0: + weight /= pairs + out.append(weight) + return out[0], out[1] + +def _own_deal(weight: np.ndarray) -> dict[tuple[int, int], np.ndarray]: + """The traverser's own deal factor per state, per shape: the pre-draw weight, or the sum over the three pre-draw holes of a post-draw state.""" + post = np.zeros(N_STATES) + for h, hole in enumerate(HOLES): + for slot in range(1, SLOTS): + thrown = DECK[slot - 1] + if thrown in hole: + continue + post[h * SLOTS + slot] = sum( + weight[state_index(tuple(c for c in hole if c != drawn) + (thrown,), ())] + for drawn in hole + ) + return {(1, 0): weight, (2, 0): weight, (2, 1): post} + +# --------------------------------------------------------------------------- +# The exact targets +# --------------------------------------------------------------------------- + +_Sigma = Sequence[np.ndarray] + +@dataclass(frozen=True, eq=False) +class ExactTargets: + """Exact regret and arrival for any table row, under a strategy read from the materialised table. + + exact = ExactTargets.build() # once per process + sigma = exact.sigma(table) # the walk's σ, per point + regret, arrival = exact.targets(sigma, seat=0, rows=rows) + + `regret` is `(n, 4)` float64, zero past each row's width; `arrival` is + `(n,)`, the expected number of visits one walk with traverser `seat` + pays the row (a probability, but for the rare walk that reaches a row + twice through two throws). + `arrivals` is the second number alone, at a fraction of the cost. + `build(holes=)` restricts what each seat may be dealt, uniformly over + the pairs that remain, as `exploitability.minidrawmaha_game(holes=)` does + and as a test's `deal` must then sample; the default is every deal. + """ + + description: _Description + weight: tuple[np.ndarray, np.ndarray] + own_deal: tuple[dict[tuple[int, int], np.ndarray], dict[tuple[int, int], np.ndarray]] + + @classmethod + def build(cls, holes: tuple[Iterable[Hole], Iterable[Hole]] | None = None) -> ExactTargets: + """The description (shared, built once) with this deal set's weights. The default is cached.""" + if holes is None: + return _full_game() + weights = _weights(holes) + return cls(_description(), weights, (_own_deal(weights[0]), _own_deal(weights[1]))) + + @property + def points(self) -> tuple[PublicPoint, ...]: + return self.description.points + + def entries_of(self, shape: tuple[int, int], position: int) -> tuple[np.ndarray, np.ndarray]: + """The physical (board row, state) entries of key `position` in `shape`'s `private_keys`.""" + keys = self.description.keys[shape] + span = slice(keys.entry_start[position], keys.entry_start[position + 1]) + return keys.entry_board[span], keys.entry_state[span] + + # -- σ ------------------------------------------------------------------ + + def sigma(self, table: PackedTable) -> tuple[np.ndarray, ...]: + """Regret matching over the whole-game table, one `(n_keys, width)` array per point, float64. + + The positive part of each row over its total, uniform where nothing + is positive: `RegretMatcher.strategy`, vectorised with one `reduceat`, + which adds a row left to right where the ledger's `sum` may pair, so + a row can differ from the walk's by one ulp. Refuses a table that is + not the whole game in the whole game's order. + """ + d = self.description + widths = table.widths() + if ( + len(table) != d.row_offset[-1] + or table.cumulative_regret.size != d.column_offset[-1] + or not np.array_equal(widths, np.repeat(d.width, np.diff(d.row_offset))) + ): + raise ValueError( + "the exact target reads the whole game's layout and was handed a table of " + f"{len(table):,} rows that is not the whole game in the whole game's order" + ) + positive = np.maximum(table.cumulative_regret, 0.0) + starts = np.cumsum(widths) - widths + totals = np.add.reduceat(positive, starts) + per_column = np.repeat(totals, widths) + uniform = np.repeat(1.0 / widths, widths) + with np.errstate(invalid="ignore", divide="ignore"): + flat = np.where(per_column > 0.0, positive / per_column, uniform) + return tuple( + flat[d.column_offset[p] : d.column_offset[p + 1]].reshape(-1, int(d.width[p])) + for p in range(len(d.points)) + ) + + # -- the targets -------------------------------------------------------- + + def targets( + self, sigma: _Sigma, *, seat: int, rows: np.ndarray + ) -> tuple[np.ndarray, np.ndarray]: + """Exact regret `(n, 4)` and arrival `(n,)` — expected visits per walk — for table `rows`, all of them `seat`'s, under `sigma`. + + A row the walk cannot reach (zero arrival) gets zero regret, not NaN; + a row of the other seat raises, as does one outside the table. + """ + rows = self._rows(seat, rows) + regret = np.zeros((rows.size, TARGET_WIDTH)) + arrival = np.zeros(rows.size) + memo: dict[tuple[int, int, int], np.ndarray] = {} + for i, (node, position, boards, states, u, total) in enumerate( + self._entries(sigma, seat, rows, memo) + ): + arrival[i] = total + if total <= 0.0: + continue + values = self._action_values(sigma, seat, node, u, states[:, None], boards)[:, 0, :] + mix = sigma[node.point][position] + baseline = values @ mix + regret[i, : len(mix)] = (values - baseline[:, None]).sum(axis=0) / total + return regret, arrival + + def arrivals(self, sigma: _Sigma, *, seat: int, rows: np.ndarray) -> np.ndarray: + """The arrival alone for table `rows`: the reach and the compatibility sum, no value below.""" + rows = self._rows(seat, rows) + memo: dict[tuple[int, int, int], np.ndarray] = {} + return np.fromiter( + (total for *_, total in self._entries(sigma, seat, rows, memo)), dtype=np.float64, count=rows.size + ) + + def reach(self, sigma: _Sigma, *, carried: int, point: int, board: int) -> np.ndarray: + """Seat `carried`'s reach at `point` on one board, over its physical states: deal weight × chance × its own σ along the path. + + What the grader carries down for the responder's opponent; here the + row's opponent is `carried = 1 − seat`. Hero's actions never enter. + """ + if carried not in SEATS: + raise ValueError(f"carried must be one of {SEATS}, got {carried!r}") + return self._reach(sigma, carried, self.description.decision_of_point[point], board, {}) + + def _rows(self, seat: int, rows: np.ndarray) -> np.ndarray: + if seat not in SEATS: + raise ValueError(f"seat must be one of {SEATS}, got {seat!r}") + d = self.description + rows = np.asarray(rows, dtype=np.int64) + if rows.ndim != 1: + raise ValueError(f"rows must be a 1-D array of table rows, got shape {rows.shape}") + if rows.size and (rows.min() < 0 or rows.max() >= d.row_offset[-1]): + raise IndexError(f"a row is outside the table's {d.row_offset[-1]:,} rows") + return rows + + def _entries(self, sigma: _Sigma, seat: int, rows: np.ndarray, memo: dict): + """Per row: its node and position, its physical entries, the weighted opponent vector at each, and the arrival.""" + d = self.description + own = self.own_deal[seat] + for row in rows.tolist(): + p = int(np.searchsorted(d.row_offset, row, side="right") - 1) + position = row - int(d.row_offset[p]) + node = d.decision_of_point[p] + if node.player != seat: + raise ValueError(f"row {row} belongs to seat {node.player}, not seat {seat}") + shape = (node.level, node.discards[seat]) + boards, states = self.entries_of(shape, position) + reach = np.stack([self._reach(sigma, 1 - seat, node, int(b), memo) for b in boards]) + u = own[shape][states][:, None] * reach + compat, _ = self._showdown(node.level, boards, states[:, None], u, shares=False) + yield node, position, boards, states, u, float(compat.sum()) + + # -- the back-solve ---------------------------------------------------- + + def _reach(self, sigma: _Sigma, carried: int, node: _Node, b: int, memo: dict) -> np.ndarray: + d = self.description + key = (carried, node.index, b) + if key in memo: + return memo[key] + if node is d.root: + return self.weight[carried] + above, edge = d.parent[node.index] + if above.kind is NodeKind.CHANCE: + level = d.levels[node.level] + if above.deals_board: + base = self._reach(sigma, carried, above, int(level.parent[b]), memo) + else: + base = self._reach(sigma, carried, above, b, memo) + out = above.prob * base * level.ok[b] + elif above.player != carried: + out = self._reach(sigma, carried, above, b, memo) + else: + base = self._reach(sigma, carried, above, b, memo) + index = d.keys[(above.level, above.discards[carried])].index[b] + valid = index >= 0 + safe = np.where(valid, index, 0) + table = sigma[above.point] + out = np.zeros(N_STATES) + for column, draw in zip(edge.actions, edge.draws, strict=True): + played = base * np.where(valid, table[safe, column], 0.0) + if draw is None: + out += played + else: + out += _push(played[None, :], draw, np.array([b]))[0] + # Memoised through the board-2 deal: the 29 level-1 points have 15 + # boards each and the 20 round-2 opening points 210 each; the rest of + # round 2 is at most two opponent steps away and is recomputed, so a + # call's memo is at most 4,635 vectors whatever rows it is handed. + if node.level <= 1 or (above.kind is NodeKind.CHANCE and above.deals_board): + memo[key] = out + return out + + # -- the value below ---------------------------------------------------- + # + # The recursion carries, per board b_i: the opponent's weighted vector + # u_i (over its 7,280 states) and a SET of hero states h_i (the same + # count on every board). Hero's own moves never touch u; a throw widens + # the set twelvefold per throw, the board-2 deal widens the boards + # fourteenfold, and values come back as one number per (board, hero + # state). Keeping u per board rather than per hero state is what makes a + # round-1 row affordable: the opponent's σ gathers and pushes below are + # done once per board, not once per hero continuation. + + def _action_values( + self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray + ) -> np.ndarray: + """Hero's weighted value below each action of `node`: `(nb, nh, width)` for boards b, hero states h (nb, nh), opponent vectors u (nb, S).""" + d = self.description + nb, nh = h.shape + values = np.zeros((nb, nh, len(node.actions))) + for edge in node.edges: + if all(draw is None for draw in edge.draws): + below = self._value(sigma, hero, edge.child, u, h, b) + for column in edge.actions: + values[..., column] = below + continue + # The throw edge: the three throws' replacement states, stacked, + # walk the one subtree below in one pass and are split after. + hole = d.hole_of[h] + fan = edge.draws[0].targets.shape[-1] + stacked = np.concatenate( + [draw.targets[b[:, None], hole] for draw in edge.draws], axis=-1 + ) # (nb, nh, 3 × 12) + below = self._value(sigma, hero, edge.child, u, stacked.reshape(nb, -1), b) + below = below.reshape(nb, nh, len(edge.draws), fan).sum(axis=-1) + for j, column in enumerate(edge.actions): + values[..., column] = below[..., j] + return values + + def _value( + self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray + ) -> np.ndarray: + """Hero's weighted expected chips from `node`, `(nb, nh)`, both seats at σ, the deck averaged.""" + d = self.description + nb, nh = h.shape + if node.kind is NodeKind.TERMINAL: + compat, share = self._showdown(node.level, b, h, u, shares=node.fold_winner is None) + if node.fold_winner is not None: + sign = 1.0 if node.fold_winner == hero else -1.0 + return sign * node.stake * compat + return node.pot * share - node.committed[hero] * compat + if node.kind is NodeKind.CHANCE: + if node.deals_board: + level = d.levels[node.child.level] + kids = level.children_of[b] # (nb, 14) + fan = kids.shape[1] + spread = node.prob * u[:, None, :] * level.ok[kids] + below = self._value( + sigma, hero, node.child, + spread.reshape(-1, N_STATES), np.repeat(h, fan, axis=0), kids.ravel(), + ) + return below.reshape(nb, fan, nh).sum(axis=1) + return self._value(sigma, hero, node.child, node.prob * u * d.levels[node.level].ok[b], h, b) + if node.player == hero: + index = d.keys[(node.level, node.discards[hero])].index[b[:, None], h] # (nb, nh) + valid = index >= 0 + mix = np.where(valid[..., None], sigma[node.point][np.where(valid, index, 0)], 0.0) + return (self._action_values(sigma, hero, node, u, h, b) * mix).sum(axis=-1) + index = d.keys[(node.level, node.discards[node.player])].index[b] # (nb, S) + valid = index >= 0 + safe = np.where(valid, index, 0) + table = sigma[node.point] + total = np.zeros((nb, nh)) + for edge in node.edges: + pushed = np.zeros_like(u) + for column, draw in zip(edge.actions, edge.draws, strict=True): + played = u * np.where(valid, table[safe, column], 0.0) + if draw is None: + pushed += played + else: + pushed += _push(played, draw, b) + total += self._value(sigma, hero, edge.child, pushed, h, b) + return total + + # -- the showdown and compatibility ------------------------------------ + + def _showdown( + self, level: int, b: np.ndarray, h: np.ndarray, u: np.ndarray, *, shares: bool + ) -> tuple[np.ndarray, np.ndarray | None]: + """Per (board i, hero state j): Σ_o u_i[o]·compat(h_ij, o) and, with `shares`, the same weighted by hero's pot share. + + Compatibility is the two card sets being disjoint — hero's hole and + thrown card against the opponent's hole and thrown card — and a hero + state that holds a card of its own board is no state and scores + zero. Folded over the opponent's HOLES: u_i summed over the 16 slots + of each hole gives the hole's total reach, and the slots of the cards + hero holds are subtracted back out, because an opponent who threw one + of hero's cards cannot be there. Shares come from the dense ranks of + both halves: ¼(2 + sign(inner) + sign(outer)). + """ + d = self.description + nb, nh = h.shape + folded = u.reshape(nb, len(HOLES), SLOTS) + total = folded.sum(axis=-1) # (nb, 455) + mine = d.holds[h].astype(np.float64).transpose(0, 2, 1) # (nb, 15, nh) + shared = d.hole_holds @ mine # (nb, 455, nh): cards an opponent hole shares with hero + excluded = folded[:, :, 1:] @ mine # (nb, 455, nh): reach of that hole's states that threw a card of hero's + terms = np.where(shared == 0.0, total[:, :, None] - excluded, 0.0) + live = (d.valid[h] & (d.levels[level].ok[b[:, None], h] > 0.0)).astype(np.float64) + compat = terms.sum(axis=1) * live + if not shares: + return compat, None + hole = d.hole_of[h] # (nb, nh) + sets = d.set_of_row[b] # (nb,) + inner = np.sign(d.inner[hole][:, None, :] - d.inner[None, :, None]) + outer = np.sign(d.outer[sets[:, None], hole][:, None, :] - d.outer[sets][:, :, None]) + share = 0.25 * (2.0 + inner + outer) + return compat, (terms * share).sum(axis=1) * live + +def _push(played: np.ndarray, draw: _DrawMap, b: np.ndarray) -> np.ndarray: + """Each pair's vector through a throw on its level-1 board: every pre-draw hole spreads over its twelve replacements.""" + m = played.shape[0] + targets = draw.targets[b] # (m, 455, 12) + offsets = (np.arange(m) * N_STATES)[:, None, None] + weights = np.repeat(played[:, draw.source], targets.shape[2], axis=1).ravel() + return np.bincount((offsets + targets).ravel(), weights=weights, minlength=m * N_STATES).reshape( + m, N_STATES + ) + +@cache +def _full_game() -> ExactTargets: + weights = _weights(None) + return ExactTargets(_description(), weights, (_own_deal(weights[0]), _own_deal(weights[1]))) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 84fe5be7..fd5c973e 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -25,7 +25,8 @@ loop. them. Loosening one to make a test pass is a finding to report, not a fix. - **Slow and heavy work is gated by an environment variable named in the skip reason**: `MINIDRAWMAHA_FULL_TABLE`, `MINIDRAWMAHA_FULL_GRADE`, - `MINIDRAWMAHA_FULL_CALIBRATION`, `MINIDRAWMAHA_FULL_DEEP`. Tests that need + `MINIDRAWMAHA_FULL_CALIBRATION`, `MINIDRAWMAHA_FULL_DEEP`, + `MINIDRAWMAHA_FULL_TARGET`. Tests that need torch `importorskip` it. The default suite must keep passing without torch, Modal, OpenSpiel or a network download. - **Committed numbers are recomputed, not transcribed.** `minidrawmaha/test_play.py` diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_exact_target.py new file mode 100644 index 00000000..9d91a1da --- /dev/null +++ b/tests/minidrawmaha/test_exact_target.py @@ -0,0 +1,622 @@ +"""The exact target: what one sampled regret row is worth on average, computed instead of sampled. + +The module under test is rung 4's boost oracle: for a table row it sums, over +every physical entry of the key and every opponent hand, the regret the walk +would bank there, weighted by how often the walk arrives. The claims, in +order: it imports the rules and never the grader; `sigma` is the ledger's +own σ row by row; the back-solved opponent reach equals the grader's carried +vector on every board of every public point, in the full game and in a +restricted-deal one (test a); at a seat's first decision the arrival IS the +grader's frequency, and a seat's first decisions sum to one; on a small +restricted game the regret and the arrival equal a world-by-world +expectation walked through `MiniState` (test b, exact form); on a wider +restricted game they match the mean and the visit count of thousands of +`Collector` walks, row by row within four standard errors and pooled (tests +b and c); rows of the wrong seat are refused and an unreachable row is zero; +and the cost per round-2 row is recorded (test d, a benchmark). + +Helpers are restated rather than imported from other test modules, as the +suite's rule has it. +""" + +from __future__ import annotations + +import ast +import os +import time +from collections.abc import Mapping +from pathlib import Path + +import numpy as np +import pytest + +from drawmaha_solver.deepcfr.collect import Collector +from drawmaha_solver.minidrawmaha import exploitability as grader +from drawmaha_solver.minidrawmaha.rules.cards import DECK, SUIT_RELABELLINGS, parse_cards, relabel +from drawmaha_solver.minidrawmaha.rules.enumeration import ( + PublicPoint, + private_keys, + public_decision_points, +) +from drawmaha_solver.minidrawmaha.rules.game import ( + Action, + InfoSet, + MiniState, + NodeKind, + canonical_picture, + random_deal, +) +from drawmaha_solver.minidrawmaha.solver.exact_target import ( + HOLES, + N_STATES, + SLOTS, + ExactTargets, + board_row, + state_index, +) +from drawmaha_solver.minidrawmaha.solver.packed_table import PackedTable + +REPO = Path(__file__).parents[2] +MODULE = REPO / "src" / "drawmaha_solver" / "minidrawmaha" / "solver" / "exact_target.py" +FULL_TARGET = os.environ.get("MINIDRAWMAHA_FULL_TARGET") == "1" + +# --------------------------------------------------------------------------- +# Helpers (restated) +# --------------------------------------------------------------------------- + +def key_on(point: PublicPoint, private) -> InfoSet: + hole, thrown, board = private + return InfoSet( + player=point.player, hole=hole, discarded=thrown, board=board, + draws=point.draws, betting=point.betting, + ) + +def shape_of(point: PublicPoint): + return private_keys(board_cards=point.board_cards, discards=point.discards) + +def random_table(seed: int) -> PackedTable: + """The whole game with N(0, 1) regrets: every row plays a different, mostly sparse σ.""" + table = PackedTable.whole_game() + table.cumulative_regret[:] = np.random.default_rng(seed).standard_normal( + table.cumulative_regret.size + ) + return table + +def positive_table(seed: int) -> PackedTable: + """The whole game with regrets in (0.05, 1.05): every action has positive probability, so every line is reachable.""" + table = PackedTable.whole_game() + table.cumulative_regret[:] = 0.05 + np.random.default_rng(seed).random(table.cumulative_regret.size) + return table + +def point_index() -> dict[tuple, int]: + """Public signature → position among the 141 points, the table's order.""" + return { + (point.board_cards, point.draws, point.betting): index + for index, point in enumerate(public_decision_points()) + } + +def grader_node_to_point(game: grader.VectorGame) -> dict[int, int]: + """The grader's own decision-node indices → table point indices, by public signature.""" + by_signature = point_index() + return { + node.index: by_signature[(node.level, node.public[0], node.public[1])] + for node in game.decisions + } + +def grader_sigma(game: grader.VectorGame, sigma) -> dict[int, np.ndarray]: + """`sigma` re-keyed by the grader's node index, as `_snapshot` would hand it to the walk.""" + to_point = grader_node_to_point(game) + return {node.index: sigma[to_point[node.index]] for node in game.decisions} + +def grader_reach(game: grader.VectorGame, sigma, *, carried: int) -> dict[int, np.ndarray]: + """The grader's forward pass, stopping at nothing: seat `carried`'s reach at every node, all boards. + + `_walk`'s chance and opponent branches verbatim (the opponent here is + `carried`); the other seat's nodes pass the reach straight down, since its + actions never enter a counterfactual reach. + """ + tables = grader_sigma(game, sigma) + out: dict[int, np.ndarray] = {} + + def forward(node, reach): + out[node.index] = reach + if node.kind is NodeKind.TERMINAL: + return + if node.kind is NodeKind.CHANCE: + child = node.child + if node.deals_board: + level = game.levels[child.level] + forward(child, node.prob * reach[level.parent] * level.ok) + else: + forward(child, node.prob * reach * game.levels[node.level].ok) + return + if node.player != carried: + for edge in node.edges: + forward(edge.child, reach) + return + keys = game.keys[(node.level, node.discards[node.player])] + valid = keys.index >= 0 + safe = np.where(valid, keys.index, 0) + table = tables[node.index] + for edge in node.edges: + pushed = np.zeros_like(reach) + for column, draw in zip(edge.actions, edge.draws, strict=True): + played = reach * np.where(valid, table[safe, column], 0.0) + if draw is None: + pushed += played + else: + grader._push(pushed, played, draw) + forward(edge.child, pushed) + + forward(game.root, game.weight[carried][None, :]) + return out + +def grader_board_row(level: int, b: int) -> int: + """My board index → the grader's batch row for the same cards (a guard against a convention drift).""" + if level == 1: + return grader.board_row((DECK[b],)) + first = b // (len(DECK) - 1) + rest = b % (len(DECK) - 1) + second = rest if rest < first else rest + 1 + return grader.board_row((DECK[first], DECK[second])) + +class RowProfile(Mapping): + """A profile over the whole game read from flat σ, float64, for the grader's functions.""" + + def __init__(self, table: PackedTable, flat: np.ndarray): + self._table, self._flat = table, flat + + def __getitem__(self, key): + _, start, width = self._table.slot_of(key) + return self._flat[start : start + width] + + def __iter__(self): + return iter(self._table) + + def __len__(self): + return len(self._table) + +# -- a restricted deal the walks can sample ------------------------------------ + +_ALLOWED: list[tuple[tuple, tuple]] = [] + +def restricted_deal(rng: np.random.Generator) -> MiniState: + """Uniform over `_ALLOWED` (seat-0 hole, seat-1 hole) pairs; module-level so the collector may pickle it.""" + return MiniState(holes=_ALLOWED[int(rng.integers(len(_ALLOWED)))]) + +def allowed_pairs(seat0, seat1) -> list[tuple[tuple, tuple]]: + return [ + (tuple(sorted(a)), tuple(sorted(b))) + for a in seat0 + for b in seat1 + if not set(a) & set(b) + ] + +# -- the world-by-world expectation ------------------------------------------ + +def world_by_world(table: PackedTable, key: InfoSet, pairs) -> tuple[np.ndarray, float]: + """E[sampled target | the walk reaches `key`] and P(reach), by enumerating every world. + + Walks `MiniState` from every allowed deal (each at 1/len(pairs)), following + every chance outcome at its probability and every action of the other + seat at its σ, and every action of the key's seat unweighted — the walk + enumerates its own actions. At the key, the exact value of each action + (`ev`: chance averaged, both seats at σ) stands in for the walk's sampled + continuation, whose expectation it is. Nothing here shares code with the + module under test: it reads σ through the ledger and the rules through + the state machine. + """ + seat = key.player + width = len(key.legal_actions()) + numerator = np.zeros(width) + denominator = 0.0 + + def ev(state: MiniState) -> float: + if state.kind is NodeKind.TERMINAL: + return state.returns()[seat] + if state.kind is NodeKind.CHANCE: + return sum(q * ev(state.apply_chance(cards)) for cards, q in state.chance_outcomes()) + sigma = table[state.infoset()].strategy() + return sum( + s * ev(state.apply(action)) + for action, s in zip(state.legal_actions(), sigma, strict=True) + if s > 0.0 + ) + + def consistent(state: MiniState) -> bool: + """Could this history still reach `key`? Its public record must be a prefix of the key's, and the seat's own cards must fit.""" + if len(state.board) > len(key.board) or len(state.draws) > len(key.draws): + return False + if state.draws != key.draws[: len(state.draws)]: + return False + for round_index, line in enumerate(state.betting): + if round_index >= len(key.betting) or key.betting[round_index][: len(line)] != line: + return False + if round_index < len(state.betting) - 1 and line != key.betting[round_index]: + return False + own = set(state.holes[seat]) | set(state.discards[seat]) + # The seat's cards so far, in the key's suit frame, must be a subset of the key's picture. + return any( + own <= set(relabel(frame, key.hole)) | set(relabel(frame, key.discarded)) + and tuple(relabel(frame, key.board))[: len(state.board)] == state.board + for frame in SUIT_RELABELLINGS + ) + + def arrive(state: MiniState, p: float) -> None: + nonlocal denominator + if state.kind is NodeKind.TERMINAL or not consistent(state): + return + if state.kind is NodeKind.CHANCE: + for cards, q in state.chance_outcomes(): + arrive(state.apply_chance(cards), p * q) + return + actions = state.legal_actions() + if state.current_player == seat: + if state.infoset() == key: + utilities = np.array([ev(state.apply(action)) for action in actions]) + sigma = table[key].strategy() + numerator[:] += p * (utilities - sigma @ utilities) + denominator += p + return # perfect recall: the key is not below itself + for action in actions: + arrive(state.apply(action), p) + return + sigma = table[state.infoset()].strategy() + for action, s in zip(actions, sigma, strict=True): + if s > 0.0: + arrive(state.apply(action), p * s) + + for holes in pairs: + arrive(MiniState(holes=holes), 1.0 / len(pairs)) + return (numerator / denominator if denominator > 0 else numerator), denominator + +def rows_at(table: PackedTable, keys) -> np.ndarray: + return np.asarray([table.row_of(key) for key in keys], dtype=np.int64) + +# --------------------------------------------------------------------------- +# Layering +# --------------------------------------------------------------------------- + +def test_the_module_imports_the_rules_and_never_the_grader_or_the_net(): + # The boost feeds the trainer; a bug shared with the grader would cancel + # and certify itself. The check is static, on the import block, so a + # review of the module starts where the root instructions say it should. + tree = ast.parse(MODULE.read_text()) + imported = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imported |= {alias.name for alias in node.names} + elif isinstance(node, ast.ImportFrom): + imported.add(node.module or "") + assert not any("exploitability" in name for name in imported), imported + assert not any("deepcfr" in name for name in imported), imported + assert not any(name.startswith("torch") for name in imported), imported + assert all( + name.startswith(("drawmaha_solver.minidrawmaha.rules", "drawmaha_solver.minidrawmaha.solver.packed_table")) + for name in imported + if name.startswith("drawmaha_solver") + ), imported + +# --------------------------------------------------------------------------- +# The description +# --------------------------------------------------------------------------- + +def test_build_is_one_object_per_deal_set_over_a_shared_description(): + full = ExactTargets.build() + assert ExactTargets.build() is full # the default is cached: the description takes seconds to build + assert len(full.points) == 141 + # The reach memo's bound, as the module states it: 29 level-1 points and + # 20 round-2 opening points (one per round-1 line and pair of draw counts). + d = full.description + opening = [ + node for node in d.decision_of_point + if node.level == 2 and d.parent[node.index][0].kind is NodeKind.CHANCE and d.parent[node.index][0].deals_board + ] + assert len(opening) == 20 + assert sum(node.level <= 1 for node in d.decision_of_point) == 29 + # Every physical state of the frame is one of (hole, nothing thrown | a card not in the hole). + assert N_STATES == len(HOLES) * SLOTS == 455 * 16 + assert state_index(parse_cards("2c 2d 2h"), ()) == 0 # the deck is ordered by rank, then suit + assert state_index(parse_cards("2c 2d 2h"), parse_cards("3c")) == 1 + DECK.index(parse_cards("3c")[0]) + assert board_row(parse_cards("2c")) == 0 and board_row(parse_cards("2d 2c")) == 14 + # Every key of every shape has its physical entries, and the entry counts + # are the frame's: boards × states the board allows. + counts = {(1, 0): 15 * 364, (2, 0): 210 * 286, (2, 1): 210 * 286 * 10} + for shape, expected in counts.items(): + n_keys = len(private_keys(board_cards=shape[0], discards=shape[1])) + total = 0 + sizes = set() + for position in range(n_keys): + boards, states = full.entries_of(shape, position) + assert boards.shape == states.shape and boards.size >= 1 + total += boards.size + sizes.add(boards.size) + assert total == expected + # A key is a suit orbit of one picture: 6 relabellings over its stabiliser. + assert sizes <= {1, 2, 3, 6} + +def test_sigma_is_the_ledgers_strategy_row_by_row(): + full = ExactTargets.build() + table = random_table(1) + sigma = full.sigma(table) + assert len(sigma) == 141 + for p, point in enumerate(public_decision_points()): + shape = shape_of(point) + assert sigma[p].shape == (len(shape), point.width) and sigma[p].dtype == np.float64 + for position in (0, len(shape) // 3, len(shape) - 1): + key = key_on(point, shape[position]) + np.testing.assert_allclose(sigma[p][position], table[key].strategy(), rtol=1e-15, atol=0) + with pytest.raises(ValueError, match="whole game"): + full.sigma(PackedTable.listed([key_on(public_decision_points()[0], shape_of(public_decision_points()[0])[0])])) + +# --------------------------------------------------------------------------- +# (a) the back-solved reach is the grader's carried reach +# --------------------------------------------------------------------------- + +RESTRICTED = ( + [parse_cards("2c 3c 5d"), parse_cards("2d 3d 5c"), parse_cards("4h 5h 6h")], + [parse_cards("2d 4h 6h"), parse_cards("3h 4c 4d"), parse_cards("2c 2d 6c"), parse_cards("3c 5h 6d")], +) + +@pytest.mark.parametrize("holes", [None, RESTRICTED], ids=["full", "restricted"]) +@pytest.mark.parametrize("carried", [0, 1]) +def test_reach_equals_the_graders_carried_reach_on_every_board(holes, carried): + exact = ExactTargets.build(holes=holes) + game = grader.minidrawmaha_game(holes=holes) + table = random_table(2) + sigma = exact.sigma(table) + reference = grader_reach(game, sigma, carried=carried) + to_point = grader_node_to_point(game) + checked = 0 + for node in game.decisions: + p = to_point[node.index] + for b in range(reference[node.index].shape[0]): + mine = exact.reach(sigma, carried=carried, point=p, board=b) + theirs = reference[node.index][grader_board_row(node.level, b)] + np.testing.assert_allclose(mine, theirs, rtol=1e-10, atol=1e-300) + checked += bool(theirs.any()) + assert checked > 1000 # most (point, board) pairs carry reach under a random table + +# --------------------------------------------------------------------------- +# arrival at a first decision is the grader's frequency +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("seat", [0, 1]) +def test_arrival_at_a_seats_first_decision_is_the_graders_frequency_and_sums_to_one(seat): + # Before a seat has acted, own reach is the deal alone, so the grader's + # `frequency` (own × compat) at its opening points is exactly the walk's + # arrival; and every walk reaches one of them. + exact = ExactTargets.build() + game = grader.minidrawmaha_game() + table = random_table(3) + sigma = exact.sigma(table) + reach = grader_reach(game, sigma, carried=1 - seat) + to_point = grader_node_to_point(game) + opening = [ + node for node in game.decisions + if node.player == seat and node.level == 1 and len(node.public[1][0]) == seat + ] + assert len(opening) == 1 + seat + total = 0.0 + for node in opening: + p = to_point[node.index] + keys = game.keys[(1, 0)] + valid = keys.index >= 0 + _, compat = game.showdown.sums(1, reach[node.index], discards=0, shares=False) + frequency = game.weight[seat][None, :] * compat + per_key = np.zeros(keys.n_keys) + np.add.at(per_key, keys.index[valid], frequency[valid]) + rows = np.arange(len(per_key)) + table.row_of(key_on(public_decision_points()[p], shape_of(public_decision_points()[p])[0])) + arrival = exact.arrivals(sigma, seat=seat, rows=rows) + assert arrival.shape == (len(rows),) + np.testing.assert_allclose(arrival, per_key, rtol=1e-10, atol=1e-300) + total += arrival.sum() + # The full computation agrees with the arrival-only path (a round-1 + # row's value walks the whole game below it, so a handful of rows). + some = rows[:: max(1, len(rows) // 4)] + regret, with_values = exact.targets(sigma, seat=seat, rows=some) + assert regret.shape == (len(some), 4) and np.isfinite(regret).all() + np.testing.assert_array_equal(with_values, arrival[:: max(1, len(rows) // 4)]) + assert total == pytest.approx(1.0, abs=1e-12) + +# --------------------------------------------------------------------------- +# (b) exact form: a world-by-world expectation on a small restricted game +# --------------------------------------------------------------------------- + +SMALL = ( + # Two suit relabellings of one hole for seat 0: one key, two physical entries. + [parse_cards("2c 3c 5d"), parse_cards("2d 3d 5c")], + # An opponent set that is NOT closed under that relabelling, so the two + # entries face different ranges and the sum over entries is load-bearing. + [parse_cards("2d 4h 6h"), parse_cards("3h 4c 4d"), parse_cards("2c 2h 6c")], +) + +def small_game_keys(seat: int) -> list[InfoSet]: + """One round-2 stood-pat row, one post-draw row, and (seat 1) one draw row, all reachable from SMALL.""" + points = public_decision_points() + picks = [] + hero = SMALL[seat][0] + if seat == 1: + # Seat 1's draw after seat 0 stood pat: the value below it expands over + # 37 hero continuations and 7 boards, which the exact walk can afford. + draw = next(p for p in points if p.player == 1 and p.is_draw_decision and p.draws[0].count == 0) + picks.append((draw, (hero, (), parse_cards("4c")))) + pat = next(p for p in points if p.player == seat and p.board_cards == 2 and p.discards == 0 and len(p.betting[1]) == seat) + picks.append((pat, (hero, (), parse_cards("4c 6d")))) + threw = next(p for p in points if p.player == seat and p.board_cards == 2 and p.discards == 1 and len(p.betting[1]) == seat) + kept = hero[:2] + (parse_cards("6d")[0],) + picks.append((threw, (kept, hero[2:], parse_cards("4c 3h")))) + return [key_on(point, canonical_key(private)) for point, private in picks] + +def canonical_key(private): + hole, thrown, board = private + return canonical_picture(tuple(sorted(hole)), tuple(sorted(thrown)), board) + +@pytest.mark.parametrize("seat", [0, 1]) +def test_regret_and_arrival_equal_a_world_by_world_expectation(seat): + pairs = allowed_pairs(*SMALL) + exact = ExactTargets.build(holes=SMALL) + table = positive_table(4) + sigma = exact.sigma(table) + keys = small_game_keys(seat) + rows = rows_at(table, keys) + regret, arrival = exact.targets(sigma, seat=seat, rows=rows) + for key, row_regret, row_arrival in zip(keys, regret, arrival, strict=True): + expected, reach = world_by_world(table, key, pairs) + assert reach > 0, str(key) + width = len(key.legal_actions()) + assert row_arrival == pytest.approx(reach, rel=1e-9, abs=0), str(key) + np.testing.assert_allclose(row_regret[:width], expected, rtol=1e-9, atol=1e-12, err_msg=str(key)) + assert not row_regret[width:].any() + # The seat-0 keys stand for two physical entries whose weights differ, + # so a representative entry would have failed above. + if seat == 0: + for key in keys: + own = [ + world_by_world(table, key, [pair for pair in pairs if pair[0] == hole])[1] + for hole in SMALL[0] + ] + assert own[0] != pytest.approx(own[1], rel=1e-3) + +# --------------------------------------------------------------------------- +# (b) and (c) sampled: thousands of walks against the exact numbers +# --------------------------------------------------------------------------- + +WIDE = ([parse_cards("2c 3c 5d")], list(HOLES)) # one hero hole for seat 0, any opponent + +def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): + # Seat 0 holds one hole, so its rows are reached often enough to test: + # 6,000 walks reach the opening row every time and the draw rows a few + # hundred times each. Per row the sample mean is within 4 s.e. of the + # exact regret and the visit count within 4 s.e. of the exact arrival; + # pooled, the mean of z² over rows sits near 1 — a bias of half a + # standard error in every row would read 1.25 and a convention bug + # (swapped reach, a dropped entry, the wrong chance factor) far above. + walks = 6_000 + _ALLOWED[:] = allowed_pairs(*WIDE) + exact = ExactTargets.build(holes=WIDE) + table = random_table(5) + sigma = exact.sigma(table) + with Collector(seed=11, workers=0, deal=restricted_deal) as collector: + collector.table.cumulative_regret[:] = table.cumulative_regret + offered = collector.collect(seat=0, t=1, walks=walks) + rows, counts = np.unique(offered.advantage_rows, return_counts=True) + enough = counts >= 40 + rows, counts = rows[enough], counts[enough] + assert rows.size >= 8 + regret, arrival = exact.targets(sigma, seat=0, rows=rows) + widths = table.widths()[rows] + z_regret, z_visits = [], [] + for row, count, width, exact_regret, exact_arrival in zip(rows, counts, widths, regret, arrival, strict=True): + samples = offered.advantage_targets[offered.advantage_rows == row][:, :width].astype(np.float64) + mean = samples.mean(axis=0) + se = samples.std(axis=0, ddof=1) / np.sqrt(count) + live = se > 0 + z = (mean - exact_regret[:width])[live] / se[live] + assert (np.abs(z) < 4).all(), (row, mean, exact_regret[:width], se) + z_regret += z.tolist() + # Visits: binomial over walks (a row reached twice in one walk is rarer than one in a thousand). + se_visits = np.sqrt(walks * exact_arrival * (1 - exact_arrival)) + zv = (count - walks * exact_arrival) / se_visits + assert abs(zv) < 4, (row, count, walks * exact_arrival) + z_visits.append(zv) + for z in (np.asarray(z_regret), np.asarray(z_visits)): + assert z.size >= 8 + assert 0.4 < np.mean(z**2) < 2.0, z + +@pytest.mark.skipif( + not FULL_TARGET, reason="200,000 whole-game walks on 8 workers, ~2 min; set MINIDRAWMAHA_FULL_TARGET=1" +) +@pytest.mark.parametrize("seat", [0, 1]) +def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat): + walks = 200_000 + exact = ExactTargets.build() + table = random_table(6) + sigma = exact.sigma(table) + with Collector(seed=12, workers=8, deal=random_deal) as collector: + collector.table.cumulative_regret[:] = table.cumulative_regret + offered = collector.collect(seat=seat, t=1, walks=walks) + rows, counts = np.unique(offered.advantage_rows, return_counts=True) + enough = counts >= 100 + rows, counts = rows[enough], counts[enough] + assert rows.size >= 50 + arrival = exact.arrivals(sigma, seat=seat, rows=rows) + z = (counts - walks * arrival) / np.sqrt(walks * arrival * (1 - arrival)) + # About 1,500 rows clear 100 visits, so the largest |z| of 1,500 fair + # draws passes 4 in about nine runs out of a hundred and 5 in one out + # of a thousand; the pooled mean of z² is the sharp test, 1 ± 0.04 at + # this many rows, and a bias of a quarter standard error in every row + # would read 1.06. + assert (np.abs(z) < 5).all(), (np.abs(z).max(), rows[np.argmax(np.abs(z))]) + assert 0.85 < np.mean(z**2) < 1.2, np.mean(z**2) + +# --------------------------------------------------------------------------- +# Refusals and the unreachable row +# --------------------------------------------------------------------------- + +def test_rows_of_the_other_seat_and_outside_the_table_are_refused(): + exact = ExactTargets.build() + table = random_table(7) + sigma = exact.sigma(table) + points = public_decision_points() + seat1 = next(i for i, point in enumerate(points) if point.player == 1) + row_of_seat_1 = table.row_of(key_on(points[seat1], shape_of(points[seat1])[0])) + with pytest.raises(ValueError, match="seat"): + exact.targets(sigma, seat=0, rows=np.array([row_of_seat_1])) + with pytest.raises(ValueError, match="seat"): + exact.targets(sigma, seat=2, rows=np.array([0])) + for bad in (-1, len(table)): + with pytest.raises(IndexError): + exact.targets(sigma, seat=0, rows=np.array([bad])) + +def test_an_unreachable_row_has_zero_arrival_and_zero_regret(): + # Seat 1 never bets or raises: every seat-0 round-1 row facing a bet is + # unreachable, and the exact target says so with zeros rather than NaN. + exact = ExactTargets.build() + table = random_table(8) + points = public_decision_points() + for p, point in enumerate(points): + if point.player == 1 and not point.is_draw_decision and Action.POT in point.actions: + column = point.actions.index(Action.POT) + for position in range(len(shape_of(point))): + _, start, width = table.slot_of(key_on(point, shape_of(point)[position])) + table.cumulative_regret[start + column] = -1.0 + if not (table.cumulative_regret[start : start + width] > 0).any(): + table.cumulative_regret[start + point.actions.index(Action.CHECK_CALL)] = 1.0 + sigma = exact.sigma(table) + facing = next( + i for i, point in enumerate(points) + if point.player == 0 and point.board_cards == 1 and not point.is_draw_decision + and point.betting[0][-1:] == (Action.POT,) + ) + rows = np.arange(len(shape_of(points[facing]))) + table.row_of(key_on(points[facing], shape_of(points[facing])[0])) + regret, arrival = exact.targets(sigma, seat=0, rows=rows) + assert not arrival.any() and not regret.any() + assert np.isfinite(regret).all() + +# --------------------------------------------------------------------------- +# (d) cost per row: a benchmark, recorded, with a loose ceiling +# --------------------------------------------------------------------------- + +def test_a_round_two_row_costs_a_few_milliseconds(): + exact = ExactTargets.build() + table = random_table(9) + sigma = exact.sigma(table) + rng = np.random.default_rng(0) + points = public_decision_points() + offsets = np.cumsum([0] + [len(shape_of(point)) for point in points]) + round_two = [p for p, point in enumerate(points) if point.player == 0 and point.board_cards == 2] + rows = np.concatenate([ + offsets[p] + rng.choice(len(shape_of(points[p])), size=10, replace=False) for p in round_two + ]) + exact.targets(sigma, seat=0, rows=rows[:5]) # warm the description + started = time.perf_counter() + exact.targets(sigma, seat=0, rows=rows) + per_row = (time.perf_counter() - started) / rows.size + print(f"\nexact target: {per_row * 1e3:.2f} ms per round-2 row over {rows.size} rows") + # The plan's figure is 1 ms a row and the measured one 0.8 ms. The + # ceiling is fifty times that: wide enough that a laptop busy with a + # training run (0.95 ms measured beside a 9-worker LCFR run) cannot + # cross it, narrow enough to catch a regression to the grader's + # whole-board update, which is about a second a row. + assert per_row < 0.050 From 8ed5c32d02e56b26a38e687897d012a997fcc2b4 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 14:58:56 -0500 Subject: [PATCH 02/16] fix(rung4): the reach memo is 66 level-1 nodes and 20 round-2 openers, 5,190 vectors, re-derived by a test; the sampled tests select rows by expected count, so a row cannot enter because it fluctuated high MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-ups on #89. The docstring counted the memo's level-1 points alone (29) where the chance nodes on the paths are memoised too (66 in all); the test now walks the ancestor closure and pins 66, 20 and 5,190. `reach()` hands out a copy. The pooled z² windows are a 4-sd band on the chi-square mean, √(2/n), instead of fixed numbers. The gated 200,000-walk test kept rows by observed count ≥ 100, which let seat 1's draw rows in only when they had fluctuated high (mean z +1.8 on two seeds); rows are now kept by expected count ≥ 80 against an observed cutoff of 40, and both seats read mean z within 0.01 of zero over thousands of rows. The module docstring names the double-visit caveat and no longer points at a plan outside the repo without saying where it is. --- .../minidrawmaha/solver/exact_target.py | 38 +++++++----- tests/minidrawmaha/test_exact_target.py | 58 +++++++++++++------ 2 files changed, 64 insertions(+), 32 deletions(-) diff --git a/src/drawmaha_solver/minidrawmaha/solver/exact_target.py b/src/drawmaha_solver/minidrawmaha/solver/exact_target.py index 66b5fcb6..23fb1b17 100644 --- a/src/drawmaha_solver/minidrawmaha/solver/exact_target.py +++ b/src/drawmaha_solver/minidrawmaha/solver/exact_target.py @@ -41,10 +41,16 @@ makes the exact row commensurable with the sampled ones: a sampled row's target is one draw of r, so its expectation is this ratio, and the boost's row weight, `K · arrival · t`, restores the total weight the K walks would -have put there. Dropping the denominator, or weighting every target 1, is -the biased "teleport" the plan rules out. The traverser's own σ enters -only inside `v(I) = ⟨σ, v(·)⟩`, the regret's baseline, as it does in the -walk. +have put there. Dropping the denominator, or weighting every target 1, +would be a "teleport" that over-weights the rare rows it lands on, which is +the bias the targeted-sampling plan (kept, like the rung-3 plan, outside +the repo on Alec's machine under `~/Desktop/Claude/poker/`) exists to +avoid. The traverser's own σ enters only inside `v(I) = ⟨σ, v(·)⟩`, the +regret's baseline, as it does in the walk. One caveat on "visits": a walk +can bank a row twice (about one traversal in a thousand, through the suit +relabelling), so `arrival` is the expected number of visits per walk, a +probability but for those, and `regret` is the mean banked sample per +visit. **Why every physical entry is summed.** A key is a suit orbit of one picture — up to six physical (board, state) pairs, fewer when the picture @@ -96,11 +102,13 @@ replacements (36 states per throw) and the second board card (14 boards): the opponent's vector is carried once per board and shared by every hero state on it, which is what keeps those within a second. The opponent's reach -is memoised per call at every level-1 point (29 of them, 15 boards each) and -at every round-2 opening point (20: one per round-1 line and pair of draw -counts, 210 boards each), 58 KB a (point, board) pair, so the memo is at -most 29 × 15 + 20 × 210 = 4,635 vectors, about 270 MB; a call over 2,240 -round-2 rows peaked at about 260 MB. `tests/minidrawmaha/test_exact_target.py` prints the +is memoised per call at every node with one board card out that lies on +the path to a decision (66: the 29 level-1 decision points and 37 chance +nodes, 15 boards each) and at every round-2 opening decision (20: one per +round-1 line and pair of draw counts, 210 boards each), 58 KB a (node, +board) pair, so the memo is at most 66 × 15 + 20 × 210 = 5,190 vectors, +302 MB — a count `tests/minidrawmaha/test_exact_target.py` re-derives +from the tree; a call over 2,240 round-2 rows peaked at about 260 MB. `tests/minidrawmaha/test_exact_target.py` prints the round-2 figure on every run. """ @@ -669,7 +677,8 @@ def reach(self, sigma: _Sigma, *, carried: int, point: int, board: int) -> np.nd """ if carried not in SEATS: raise ValueError(f"carried must be one of {SEATS}, got {carried!r}") - return self._reach(sigma, carried, self.description.decision_of_point[point], board, {}) + # A copy: at the root the vector is the deal weight itself. + return self._reach(sigma, carried, self.description.decision_of_point[point], board, {}).copy() def _rows(self, seat: int, rows: np.ndarray) -> np.ndarray: if seat not in SEATS: @@ -731,10 +740,11 @@ def _reach(self, sigma: _Sigma, carried: int, node: _Node, b: int, memo: dict) - out += played else: out += _push(played[None, :], draw, np.array([b]))[0] - # Memoised through the board-2 deal: the 29 level-1 points have 15 - # boards each and the 20 round-2 opening points 210 each; the rest of - # round 2 is at most two opponent steps away and is recomputed, so a - # call's memo is at most 4,635 vectors whatever rows it is handed. + # Memoised through the board-2 deal: the 66 level-1 nodes on the + # paths (decisions and chance nodes) have 15 boards each and the 20 + # round-2 opening decisions 210 each; the rest of round 2 is at most + # two opponent steps away and is recomputed, so a call's memo is at + # most 5,190 vectors whatever rows it is handed. if node.level <= 1 or (above.kind is NodeKind.CHANCE and above.deals_board): memo[key] = out return out diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_exact_target.py index 9d91a1da..a7adbd93 100644 --- a/tests/minidrawmaha/test_exact_target.py +++ b/tests/minidrawmaha/test_exact_target.py @@ -305,15 +305,22 @@ def test_build_is_one_object_per_deal_set_over_a_shared_description(): full = ExactTargets.build() assert ExactTargets.build() is full # the default is cached: the description takes seconds to build assert len(full.points) == 141 - # The reach memo's bound, as the module states it: 29 level-1 points and - # 20 round-2 opening points (one per round-1 line and pair of draw counts). + # The reach memo's bound, as the module states it: every node with one + # board card out on the path from a decision to the root, and every + # round-2 opening decision (one per round-1 line and pair of draw counts). d = full.description - opening = [ - node for node in d.decision_of_point - if node.level == 2 and d.parent[node.index][0].kind is NodeKind.CHANCE and d.parent[node.index][0].deals_board - ] - assert len(opening) == 20 - assert sum(node.level <= 1 for node in d.decision_of_point) == 29 + memoised = {} + for node in d.decision_of_point: + while node is not d.root: + above, _ = d.parent[node.index] + if node.level <= 1 or (above.kind is NodeKind.CHANCE and above.deals_board): + memoised[node.index] = node + node = above + level_one = [node for node in memoised.values() if node.level <= 1] + opening = [node for node in memoised.values() if node.level == 2] + assert len(level_one) == 66 and sum(node.kind is NodeKind.DECISION for node in level_one) == 29 + assert len(opening) == 20 and all(node.kind is NodeKind.DECISION for node in opening) + assert len(level_one) * 15 + len(opening) * 210 == 5_190 # Every physical state of the frame is one of (hole, nothing thrown | a card not in the hole). assert N_STATES == len(HOLES) * SLOTS == 455 * 16 assert state_index(parse_cards("2c 2d 2h"), ()) == 0 # the deck is ordered by rank, then suit @@ -503,8 +510,14 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): rows, counts = np.unique(offered.advantage_rows, return_counts=True) enough = counts >= 40 rows, counts = rows[enough], counts[enough] - assert rows.size >= 8 regret, arrival = exact.targets(sigma, seat=0, rows=rows) + # Rows are kept by their EXPECTED count, 80 or more, so the observed + # cutoff of 40 sits 4.5 standard errors below every kept row's mean and + # the selection cannot bias z: a row chosen by its observed count alone + # is there because it fluctuated high. + fair = walks * arrival >= 80 + rows, counts, regret, arrival = rows[fair], counts[fair], regret[fair], arrival[fair] + assert rows.size >= 8 widths = table.widths()[rows] z_regret, z_visits = [], [] for row, count, width, exact_regret, exact_arrival in zip(rows, counts, widths, regret, arrival, strict=True): @@ -522,7 +535,9 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): z_visits.append(zv) for z in (np.asarray(z_regret), np.asarray(z_visits)): assert z.size >= 8 - assert 0.4 < np.mean(z**2) < 2.0, z + # The mean of n squared standard normals has mean 1 and standard + # deviation √(2/n); this is a 4-sd band on it, (0.4, 1.6) at n = 10. + assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / z.size), (z.size, np.mean(z**2)) @pytest.mark.skipif( not FULL_TARGET, reason="200,000 whole-game walks on 8 workers, ~2 min; set MINIDRAWMAHA_FULL_TARGET=1" @@ -537,18 +552,25 @@ def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat collector.table.cumulative_regret[:] = table.cumulative_regret offered = collector.collect(seat=seat, t=1, walks=walks) rows, counts = np.unique(offered.advantage_rows, return_counts=True) - enough = counts >= 100 + enough = counts >= 40 rows, counts = rows[enough], counts[enough] - assert rows.size >= 50 arrival = exact.arrivals(sigma, seat=seat, rows=rows) + # Kept by EXPECTED count, 80 or more: the observed cutoff of 40 is then + # 4.5 standard errors below every kept row's mean, so the selection + # cannot bias z. (Selecting on the observed count alone put seat 1's + # draw rows, whose means sit near 80, at a mean z of +1.8 on two seeds: + # the rows that cleared the cutoff were the ones that fluctuated high. + # Seat 1's commonest rows, its two opening points, expect about 110.) + fair = walks * arrival >= 80 + rows, counts, arrival = rows[fair], counts[fair], arrival[fair] + assert rows.size >= 50 z = (counts - walks * arrival) / np.sqrt(walks * arrival * (1 - arrival)) - # About 1,500 rows clear 100 visits, so the largest |z| of 1,500 fair - # draws passes 4 in about nine runs out of a hundred and 5 in one out - # of a thousand; the pooled mean of z² is the sharp test, 1 ± 0.04 at - # this many rows, and a bias of a quarter standard error in every row - # would read 1.06. + # Of about a thousand fair rows, the largest |z| passes 4 in about six + # runs out of a hundred and 5 in one out of a thousand; the pooled mean + # of z² is the sharp test (mean 1, standard deviation √(2/n)), and a + # bias of a quarter standard error in every row would read 1.06. assert (np.abs(z) < 5).all(), (np.abs(z).max(), rows[np.argmax(np.abs(z))]) - assert 0.85 < np.mean(z**2) < 1.2, np.mean(z**2) + assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / z.size), (z.size, np.mean(z**2)) # --------------------------------------------------------------------------- # Refusals and the unreachable row From 0712aa1e5775263331e12e9049e455b86ade5c38 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 15:24:43 -0500 Subject: [PATCH 03/16] =?UTF-8?q?fix(rung4):=20the=20sampled=20tests=20pin?= =?UTF-8?q?=20the=20pooled=20mean=20of=20z=20at=204/=E2=88=9An=20beside=20?= =?UTF-8?q?the=20mean=20of=20z=C2=B2,=20and=20cap=20a=20single=20row=20at?= =?UTF-8?q?=20six=20standard=20errors=20=E2=80=94=20the=20deal=20alone=20d?= =?UTF-8?q?ealt=20one=20picture=20292=20times=20against=20220=20at=20seed?= =?UTF-8?q?=2012?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pooled mean is the sharp bias test (a tenth of a standard error in every row is caught at 1,600 rows); the per-row cap is loose because of about 3,000 rows one passes 5 in a run out of a thousand, and the deal itself did it at seed 12 in a count that involves nothing of this module. Run with MINIDRAWMAHA_FULL_TARGET=1: both seats pass. --- tests/minidrawmaha/test_exact_target.py | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_exact_target.py index a7adbd93..70ab2a71 100644 --- a/tests/minidrawmaha/test_exact_target.py +++ b/tests/minidrawmaha/test_exact_target.py @@ -535,8 +535,10 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): z_visits.append(zv) for z in (np.asarray(z_regret), np.asarray(z_visits)): assert z.size >= 8 - # The mean of n squared standard normals has mean 1 and standard - # deviation √(2/n); this is a 4-sd band on it, (0.4, 1.6) at n = 10. + # Pooled: the mean of n fair z has standard deviation 1/√n and the + # mean of n squared ones has mean 1 and standard deviation √(2/n); + # both are 4-sd bands, (−1.3, 1.3) and (0.4, 1.6) at n = 10. + assert abs(np.mean(z)) < 4 / np.sqrt(z.size), (z.size, np.mean(z)) assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / z.size), (z.size, np.mean(z**2)) @pytest.mark.skipif( @@ -565,11 +567,17 @@ def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat rows, counts, arrival = rows[fair], counts[fair], arrival[fair] assert rows.size >= 50 z = (counts - walks * arrival) / np.sqrt(walks * arrival * (1 - arrival)) - # Of about a thousand fair rows, the largest |z| passes 4 in about six - # runs out of a hundred and 5 in one out of a thousand; the pooled mean - # of z² is the sharp test (mean 1, standard deviation √(2/n)), and a - # bias of a quarter standard error in every row would read 1.06. - assert (np.abs(z) < 5).all(), (np.abs(z).max(), rows[np.argmax(np.abs(z))]) + # The sharp tests are pooled: the mean of z over n fair rows has standard + # deviation 1/√n, so a bias of a tenth of a standard error in every row + # is caught at n ≈ 1,600; the mean of z² has mean 1 and standard + # deviation √(2/n). The per-row cap is loose on purpose: of some 3,000 + # rows the largest |z| passes 5 in a run out of a thousand, and the deal + # alone can do it — at seed 12 one seat-1 picture (3c 3d 4h with 2c on + # the board, six relabellings) was dealt 292 times against an expected + # 220, 4.9 standard errors, in a count that involves no σ and nothing of + # this module, and 221, 219, 222 times on three other seeds. + assert (np.abs(z) < 6).all(), (np.abs(z).max(), rows[np.argmax(np.abs(z))]) + assert abs(np.mean(z)) < 4 / np.sqrt(z.size), (z.size, np.mean(z)) assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / z.size), (z.size, np.mean(z**2)) # --------------------------------------------------------------------------- From 918a76cfb87d545f9733684cd49445db2bc8504b Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 15:27:41 -0500 Subject: [PATCH 04/16] =?UTF-8?q?fix(rung4):=20one=20pooled-fairness=20hel?= =?UTF-8?q?per=20for=20the=20sampled=20tests,=20with=20the=20band's=20assu?= =?UTF-8?q?mptions=20written=20beside=20it=20=E2=80=94=20the=20"caught=20a?= =?UTF-8?q?t=201,600=20rows"=20line=20was=20a=20half-truth;=20the=20band?= =?UTF-8?q?=20is=20reached=20there=20and=20reliable=20at=20about=203,000?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- tests/minidrawmaha/test_exact_target.py | 45 ++++++++++++++++--------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_exact_target.py index 70ab2a71..5afe802e 100644 --- a/tests/minidrawmaha/test_exact_target.py +++ b/tests/minidrawmaha/test_exact_target.py @@ -273,6 +273,26 @@ def arrive(state: MiniState, p: float) -> None: def rows_at(table: PackedTable, keys) -> np.ndarray: return np.asarray([table.row_of(key) for key in keys], dtype=np.int64) +def assert_pooled_fair(z: np.ndarray) -> None: + """The z of many rows, pooled: mean within 4/√n of 0, mean of z² within 4√(2/n) of 1. + + The mean of z is the sharp bias test: at n rows its standard deviation + is 1/√n, so a bias of a tenth of a standard error in every row reaches + the band at n ≈ 1,600 and is caught reliably by about 3,000 rows. The + widths take the z's as independent and normal, which they are not + quite: rows on one path share walks (a positive correlation, so the + regret band is a little narrower than four true standard deviations), + visit counts over rows come from one multinomial (negative, so that + band is conservative), and a z built from a plug-in standard error over + 40-odd samples has heavier tails than normal. The seeds are fixed, so + the runs are deterministic, and the thresholds were checked at those + seeds and on several others; they are a calibrated regression gate, + not a distributional guarantee. + """ + n = z.size + assert abs(np.mean(z)) < 4 / np.sqrt(n), (n, np.mean(z)) + assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / n), (n, np.mean(z**2)) + # --------------------------------------------------------------------------- # Layering # --------------------------------------------------------------------------- @@ -535,11 +555,7 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): z_visits.append(zv) for z in (np.asarray(z_regret), np.asarray(z_visits)): assert z.size >= 8 - # Pooled: the mean of n fair z has standard deviation 1/√n and the - # mean of n squared ones has mean 1 and standard deviation √(2/n); - # both are 4-sd bands, (−1.3, 1.3) and (0.4, 1.6) at n = 10. - assert abs(np.mean(z)) < 4 / np.sqrt(z.size), (z.size, np.mean(z)) - assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / z.size), (z.size, np.mean(z**2)) + assert_pooled_fair(z) # (−1.3, 1.3) and (0.4, 1.6) at n = 10 @pytest.mark.skipif( not FULL_TARGET, reason="200,000 whole-game walks on 8 workers, ~2 min; set MINIDRAWMAHA_FULL_TARGET=1" @@ -567,18 +583,15 @@ def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat rows, counts, arrival = rows[fair], counts[fair], arrival[fair] assert rows.size >= 50 z = (counts - walks * arrival) / np.sqrt(walks * arrival * (1 - arrival)) - # The sharp tests are pooled: the mean of z over n fair rows has standard - # deviation 1/√n, so a bias of a tenth of a standard error in every row - # is caught at n ≈ 1,600; the mean of z² has mean 1 and standard - # deviation √(2/n). The per-row cap is loose on purpose: of some 3,000 - # rows the largest |z| passes 5 in a run out of a thousand, and the deal - # alone can do it — at seed 12 one seat-1 picture (3c 3d 4h with 2c on - # the board, six relabellings) was dealt 292 times against an expected - # 220, 4.9 standard errors, in a count that involves no σ and nothing of - # this module, and 221, 219, 222 times on three other seeds. + # The sharp tests are the pooled ones (`assert_pooled_fair`). The + # per-row cap is loose on purpose: of some 3,000 rows the largest |z| + # passes 5 in a run out of a thousand, and the deal alone can do it — at + # seed 12 one seat-1 picture (3c 3d 4h with 2c on the board, six + # relabellings) was dealt 292 times against an expected 220, 4.9 + # standard errors, in a count that involves no σ and nothing of this + # module, and 221, 219, 222 times on three other seeds. assert (np.abs(z) < 6).all(), (np.abs(z).max(), rows[np.argmax(np.abs(z))]) - assert abs(np.mean(z)) < 4 / np.sqrt(z.size), (z.size, np.mean(z)) - assert abs(np.mean(z**2) - 1.0) < 4 * np.sqrt(2 / z.size), (z.size, np.mean(z**2)) + assert_pooled_fair(z) # --------------------------------------------------------------------------- # Refusals and the unreachable row From 30b7951f42c1e9cfb2cc16247a6c2f2524c8ed38 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 15:30:30 -0500 Subject: [PATCH 05/16] =?UTF-8?q?fix(rung4):=20the=20pooled=20band's=20sma?= =?UTF-8?q?ll-n=20width=20stated=20as=20it=20is=20=E2=80=94=201=20=C2=B1?= =?UTF-8?q?=201.79=20at=20n=20=3D=2010,=20biting=20from=20n=20=E2=89=88=20?= =?UTF-8?q?45=20=E2=80=94=20and=20the=20arrival=20seeds=20checked=20named?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- tests/minidrawmaha/test_exact_target.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_exact_target.py index 5afe802e..90a268d0 100644 --- a/tests/minidrawmaha/test_exact_target.py +++ b/tests/minidrawmaha/test_exact_target.py @@ -285,9 +285,12 @@ def assert_pooled_fair(z: np.ndarray) -> None: visit counts over rows come from one multinomial (negative, so that band is conservative), and a z built from a plug-in standard error over 40-odd samples has heavier tails than normal. The seeds are fixed, so - the runs are deterministic, and the thresholds were checked at those - seeds and on several others; they are a calibrated regression gate, - not a distributional guarantee. + the runs are deterministic. The arrival bands were checked at the + gated test's seed 12 for both seats and at seeds 13, 14, 15, 16, 17, + 18 and 19 for seat 1 (2026-11-05, mean z within 0.01 of zero on every + fair selection); they are a calibrated regression gate, not a + distributional guarantee. At small n the z² band is wide: 1 ± 1.79 at + n = 10, so its low side bites only from n ≈ 45 (1 ± 0.84). """ n = z.size assert abs(np.mean(z)) < 4 / np.sqrt(n), (n, np.mean(z)) @@ -555,7 +558,7 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): z_visits.append(zv) for z in (np.asarray(z_regret), np.asarray(z_visits)): assert z.size >= 8 - assert_pooled_fair(z) # (−1.3, 1.3) and (0.4, 1.6) at n = 10 + assert_pooled_fair(z) @pytest.mark.skipif( not FULL_TARGET, reason="200,000 whole-game walks on 8 workers, ~2 min; set MINIDRAWMAHA_FULL_TARGET=1" From 23cd46473fd3e69d69f7885b036781ea8d28274c Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 15:32:15 -0500 Subject: [PATCH 06/16] =?UTF-8?q?fix(rung4):=20the=20z=C2=B2=20band's=20lo?= =?UTF-8?q?w=20side=20bites=20from=20n=20=3D=2033,=20where=20its=20half-wi?= =?UTF-8?q?dth=20drops=20under=201,=20not=20from=2045?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- tests/minidrawmaha/test_exact_target.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_exact_target.py index 90a268d0..4a81c200 100644 --- a/tests/minidrawmaha/test_exact_target.py +++ b/tests/minidrawmaha/test_exact_target.py @@ -290,7 +290,8 @@ def assert_pooled_fair(z: np.ndarray) -> None: 18 and 19 for seat 1 (2026-11-05, mean z within 0.01 of zero on every fair selection); they are a calibrated regression gate, not a distributional guarantee. At small n the z² band is wide: 1 ± 1.79 at - n = 10, so its low side bites only from n ≈ 45 (1 ± 0.84). + n = 10; its low side bites once the half-width drops under 1, from + n = 33 (1 ± 0.98), and is 1 ± 0.84 at n = 45. """ n = z.size assert abs(np.mean(z)) < 4 / np.sqrt(n), (n, np.mean(z)) From 8122ccd65bc7a258d8e47e7db840c1997d11b5eb Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 17:32:42 -0500 Subject: [PATCH 07/16] feat(rung4): every memory row carries a float32 weight, one unless a caller says otherwise; a file without the column loads as ones PR 2 of the targeted-sampling plan, first of three commits. The reservoir's row grows from (row, t, target) to (row, t, target, weight): 24 to 28 bytes, a 50M-row memory 1.2 to 1.4 GB. The weight rides its row's slot through Algorithm R's eviction, save and load; merge without a weight stamps ones; a file with no weight key is from before the column and loads with ones. Refused by name: a negative, non-finite, mis-shaped weight on merge, and a weight array of the wrong length or dtype on load. The learner seam still takes three arrays, so tests/deepcfr/test_train.py is red until the next commit, which threads the column through it. --- ...-05-targeted-sampling-pr2-weight-column.md | 159 ++++++++++++++++++ scripts/deepcfr/modal_rung4.py | 2 +- src/drawmaha_solver/deepcfr/AGENTS.md | 11 +- src/drawmaha_solver/deepcfr/__init__.py | 5 +- src/drawmaha_solver/deepcfr/reservoir.py | 112 +++++++++--- tests/deepcfr/test_reservoir.py | 92 +++++++++- 6 files changed, 339 insertions(+), 42 deletions(-) create mode 100644 docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md diff --git a/docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md b/docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md new file mode 100644 index 00000000..9732210d --- /dev/null +++ b/docs/superpowers/plans/2026-11-05-targeted-sampling-pr2-weight-column.md @@ -0,0 +1,159 @@ +# Targeted sampling, PR 2: the weight column — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** every memory row carries a float32 `weight`; the refit's loss and the oracle weigh a row by `t × weight`; weight ≡ 1 reproduces today's run bit for bit, and a memory file written before the column existed loads with weights of one. + +**Architecture:** the column is a fourth array in `Reservoir`, riding the same slot as its row through Algorithm R's eviction, save and load; `contents()` grows from three arrays to four and that tuple flows unchanged through `step` into `Learner.fit`. The loss multiplies `t` by `weight` before the batch-mean normalisation; the oracle multiplies each sample by its weight before LCFR's per-t′ bank. Nothing consumes weights other than one yet: `collect.rows_of` keeps producing unweighted rows, `merge` without a weight stamps ones, and PR 4's boost stage will be the first caller to pass the column. + +**Tech Stack:** Python 3.12+, NumPy; torch only inside `fit.py`, `policy.py` bodies and the torch-gated tests. + +**Spec:** the targeted-sampling plan, outside the repo, `/Users/alec/Desktop/Claude/poker/targeted-sampling-plan/targeted-sampling-plan.md` — §3 (why an exact row's weight is `(1 − α) · K · arrival · t` and why the expected weight must be unchanged) and the appendix sections "PR 2 — weight column" (the contract) and "PR 4 — the boost stage" (the consumer). + +## Global Constraints + +- The oracle pin `tests/deepcfr/test_train.py::test_the_oracle_loop_is_lcfr_bit_for_bit` passes unchanged. The oracle banks `Σ t′ · (w · r)` per t′ in increasing order with `np.add.at` in memory order; with w ≡ 1 the multiplication is exact and the bits do not move. +- The loss is `(t · w) / mean(t · w)` per row; with w ≡ 1 this is `t / mean(t)` bit for bit (`t · 1.0 = t` in float32). The weight is a sample weight, never a divisor of the target: `pot_scaled` composes with it untouched. +- `weight ≥ 0` and finite; a row of weight 0 is inert in the loss and in the oracle's bank; a weight of 2 equals the same row offered twice (oracle). Rows stay in offer order. +- A reservoir file with no `weight` key loads with ones (a saved run resumes as the run it was); a file with a `weight` of the wrong length or dtype is refused by name. +- No new `DeepConfig` field: nothing to add to `ADDED_LATER` or `SAMPLE_FIELDS`. The resume state's file set does not change; the weight lives inside each memory's `.npz`. +- torch stays optional; the default suite passes without it. `uv run pytest` is the gate (CI runs no tests). +- Every prose claim that says a row is 24 bytes, a 50M-row memory is 1.2 GB, or the row is `(row, t, target)` is corrected in the same change: `reservoir.py`, `fit.py`, `train.py`, `policy.py`, `deepcfr/__init__.py`, `deepcfr/AGENTS.md`, `scripts/deepcfr/modal_rung4.py:155`, `tests/deepcfr/test_reservoir.py`. +- Commit subjects `feat(rung4): `; branch from PR 1's head (`23cd464`), never rebased onto main. No `git stash`, no Modal. + +## Review Focus + +1. A minibatch whose every row has weight 0 (small memory, α = 0 in PR 4): the loss must be finite, not 0/0 = NaN that ends the fit as "diverged" — pinned in Task 2 (`test_a_batch_of_weightless_rows_has_a_finite_loss`). +2. A memory whose every row has weight 0: refused by name before the fit, like the empty memory — Task 2 (`test_fit_refuses_bad_weights`). +3. A weight landing in the wrong slot during eviction (the column assigned with a different index than the row's) — Task 1 (`test_a_weight_rides_its_row_through_eviction`). +4. A `weight` array saved as float64 or of the wrong length being poured into the float32 column in silence — Task 1 (`test_load_refuses_a_weight_of_the_wrong_length_or_type`). +5. The grader's Π fit and the sweep script unpacking three arrays from a `contents()` that now returns four — Task 2 (`fit_policy` takes the fourth positionally; `pi_loss_sweep.py` unpacks four). + +--- + +### Task 1: the column in the reservoir + +**Files:** +- Modify: `src/drawmaha_solver/deepcfr/reservoir.py` (module docstring, `_FILE_KEYS`, `Reservoir.__init__`, `merge`, `contents`, `save`, `load`, `_validate_offer`) +- Modify: `src/drawmaha_solver/deepcfr/AGENTS.md:72-73`, `src/drawmaha_solver/deepcfr/__init__.py:16-18`, `scripts/deepcfr/modal_rung4.py:155` +- Test: `tests/deepcfr/test_reservoir.py` + +**Interfaces:** +- Produces: `Reservoir.merge(rows, targets, *, t, rng, weight=None)` — `weight` (n,) float, finite, ≥ 0; `None` stamps every row 1.0. `Reservoir.contents() -> (rows int32, t int32, targets float32 (n, 4), weight float32 (n,))`, read-only views. `save` writes a `weight` key; `load` reads it when present, else ones; `_WEIGHT_KEY = "weight"` beside `_FILE_KEYS`, which stays the required set. + +- [ ] **Step 1: Write the failing tests** in `tests/deepcfr/test_reservoir.py` + +```python +def test_rows_offered_without_a_weight_weigh_one(): + rows, t, kept, weight = filled_memory().contents() + assert weight.dtype == np.float32 and weight.tolist() == [1.0] * 5 + +def test_a_weight_rides_its_row_through_eviction(): + memory = Reservoir(capacity=5) + memory.merge(np.arange(3), targets(3), t=1, rng=np.random.default_rng(0), weight=np.array([0.5, 1.0, 1.5])) + memory.merge(np.arange(4) + 10, targets(4), t=2, rng=FixedDraws([0, 7]), weight=np.array([2.0, 3.0, 4.0, 5.0])) + rows, _, _, weight = memory.contents() + assert rows.tolist() == [12, 1, 2, 10, 11] + assert weight.tolist() == [4.0, 1.0, 1.5, 2.0, 3.0] + +def test_merge_refuses_bad_weights(): + # negative, NaN, wrong length, 2-D: each a ValueError matching "weight" + +def test_a_file_written_before_the_weight_existed_loads_with_ones(tmp_path): + # np.savez with rows/t/targets/offered/capacity only → load → weight == ones(size), and the next merge still works + +def test_load_refuses_a_weight_of_the_wrong_length_or_type(tmp_path): + # weight of length size − 1 → ValueError matching the file name; weight float64 → "weight must be float32" +``` + +Extend `test_contents_are_read_only_and_feed_the_fit` (four views; `weight` float32 `(5,)`, read-only) and `test_never_exceeds_capacity_and_counts_every_offer` (weights given as `rows % 100` so the fourth column moves with the other three). The round trip in `test_save_is_atomic_and_load_refuses_a_foreign_file` already zips `contents()` strictly. + +- [ ] **Step 2: Run them to make sure they fail**: `uv run pytest tests/deepcfr/test_reservoir.py -q -p no:cacheprovider` — the new tests fail on `TypeError` (unexpected `weight`) and on unpacking three values into four. + +- [ ] **Step 3: Implement** — a fourth `np.zeros(capacity, float32)` array; `_validate_offer` grows a `weight` parameter and returns it as float32 (ones when `None`); `merge` assigns `self._weight[slot[winners]] = weight[winners]`; `contents` returns four views; `save` writes `weight=`; `load` reads `saved[_WEIGHT_KEY]` when present (shape `(size,)`, dtype float32, both refused by name) else `np.ones(size, float32)`. Docstrings: the row is `(row, t, target, weight)`, 28 bytes, a 50M-row memory 1.4 GB; the weight's meaning (how many sampled rows this row stands for; 1 for every row the walk offers, PR 4's exact rows carry `(1 − α) · K · arrival`); why a missing key reads as one. + +- [ ] **Step 4: Run the reservoir tests and the default rung-4 suite**: `uv run pytest tests/deepcfr -q -p no:cacheprovider` — `test_train.py` fails at `Spy.fit` and `test_fit_receives_the_memory_as_it_stands_after_the_merge` (four arrays into a three-way zip) until Task 3; everything else passes. + +- [ ] **Step 5: Commit** `feat(rung4): every memory row carries a float32 weight, one unless a caller says otherwise; a file without the column loads as ones` + +### Task 2: the loss weighs a row by t × weight + +**Files:** +- Modify: `src/drawmaha_solver/deepcfr/fit.py` (module docstring, `batch_loss`, `_weighted_loss`, `fit_from_scratch`, `_fit_on_device`) +- Modify: `src/drawmaha_solver/deepcfr/policy.py` (`fit_policy`, its docstring at line 24), `scripts/deepcfr/pi_loss_sweep.py:225` +- Test: `tests/deepcfr/test_fit.py` (torch-gated) + +**Interfaces:** +- Consumes: `Reservoir.contents()` four-tuple (Task 1). +- Produces: `batch_loss(net, rows, t, targets, weight=None) -> Tensor`; `_weighted_loss(outputs, gather, targets, t, weight) -> Tensor` with `scale = t * weight; (scale / scale.mean().clamp(min=tiny) * error).mean()`; `fit_from_scratch(rows, t, targets, weight=None, *, seed, config)` refusing a `weight` that is not 1-D, not the memory's length, non-finite, negative, or with no positive entry ("no row carries weight: nothing to fit"); `_fit_on_device(rows, t, targets, weight, folded, config, *, device)`; `fit_policy(rows, t, targets, weight=None, *, seed, iteration, config)`. + +- [ ] **Step 1: Write the failing tests** in `tests/deepcfr/test_fit.py` + +```python +def test_weight_one_is_the_unweighted_fit_bit_for_bit(): + rows, t, targets = real_rows(300, seed=4) + plain = fit_from_scratch(rows, t, targets, seed=(0, 1, 0), config=SMALL) + ones = fit_from_scratch(rows, t, targets, np.ones(300, np.float32), seed=(0, 1, 0), config=SMALL) + for a, b in zip(plain.parameters(), ones.parameters(), strict=True): + assert torch.equal(a, b) + +def test_a_row_of_weight_zero_is_inert(): + # one row twice: +1 at weight 0, −1 at weight 1, both t = 1 → prediction −1 within 0.05 + +def test_the_loss_weighs_a_row_by_t_times_weight(): + # one row twice: +1 at (t = 1, w = 3) and −1 at (t = 3, w = 1): (3·1 − 3·1)/6 = 0; swap to w = (1, 1): (1 − 3)/4 = −0.5 + +def test_a_batch_of_weightless_rows_has_a_finite_loss(): + # batch_loss with weight = zeros → 0.0, finite, backward finite + +def test_fit_refuses_bad_weights(): + # negative → "weight"; NaN → "finite"; length n − 1 → "length"; all zero → "nothing to fit"; 2-D → "1-D" +``` + +Pass `weight=np.ones(...)` through `test_the_device_loop_on_the_cpu_is_the_cpu_fit_bit_for_bit`'s `_fit_on_device` call. + +- [ ] **Step 2: Run to make sure they fail** — torch is absent on this Mac, so run under the `deep` extra once: `uv run --extra deep pytest tests/deepcfr/test_fit.py -q -p no:cacheprovider`. Expected: the new tests fail on `TypeError`. + +- [ ] **Step 3: Implement** the signatures above. In `fit_from_scratch`, `weight = np.ones(len(rows), np.float32)` when `None`, else `np.asarray(weight, dtype=np.float32)` validated after `t`. In `NetLearner` (Task 3) and the cpu loop, index `weight[picked]` beside `t[picked]`. Docstring: the loss paragraph says `t · weight`; the device paragraph counts the weight in the copy. + +- [ ] **Step 4: Run** `uv run --extra deep pytest tests/deepcfr/test_fit.py tests/deepcfr/test_policy.py -q -p no:cacheprovider` → PASS; then `uv sync` to drop torch again and `uv run pytest tests/deepcfr/test_fit.py -q` → skipped. + +- [ ] **Step 5: Commit** `feat(rung4): the refit weighs a row by t × weight; weight one is today's loss bit for bit and a row of weight zero is inert` + +### Task 3: the learner seam and the oracle + +**Files:** +- Modify: `src/drawmaha_solver/deepcfr/train.py` (module docstring lines 15, 36–37, 55–65; `Learner.fit`; `step`'s docstring; `NetLearner.fit`; `OracleModel`; `OracleLearner.fit/write/save/load`) +- Modify: `src/drawmaha_solver/deepcfr/AGENTS.md` (the oracle-pin rule names the weight) +- Test: `tests/deepcfr/test_train.py`, `tests/deepcfr/test_train_torch.py` + +**Interfaces:** +- Consumes: Task 1's `contents()`, Task 2's `fit_from_scratch` and `batch_loss`. +- Produces: `Learner.fit(rows, t, targets, weight, *, seed) -> Fitted` (weight required at the seam, so no learner can ignore the column by accident); `OracleModel(rows, t, targets, weight)` with `weight: float32 (n,)`; `OracleLearner.write` banks `summed[touched] += int(t) * totals` where `totals` sums `values * repeat(weight, width)` in float64. + +- [ ] **Step 1: Write the failing tests** in `tests/deepcfr/test_train.py` + +```python +def test_the_oracle_weighs_a_row_by_its_weight_and_a_weight_of_two_is_the_row_twice(): + # three OracleModels over one real row r = (0.5, −0.25, 0, 0) at t = 3: + # twice at weight 1 == once at weight 2 (array_equal) + # once at weight 1 + once at weight 0 == once at weight 1 (array_equal) +def test_oracle_models_round_trip # extend names to ("rows", "t", "targets", "weight") +def test_an_oracle_model_saved_without_a_weight_loads_with_ones(tmp_path) +``` + +Update `Spy.fit(self, rows, t, targets, weight, *, seed)` to record four arrays; `test_fit_receives_the_memory_as_it_stands_after_the_merge` zips four. In `test_train_torch.py` pass `np.ones(n, np.float32)` as the fourth positional at lines 110, 112, 149, 154, 159 and in the `spy` there (`spy(rows, t, targets, weight=None, *, seed, config)`). + +- [ ] **Step 2: Run to make sure they fail**: `uv run pytest tests/deepcfr/test_train.py -q -p no:cacheprovider -x`. + +- [ ] **Step 3: Implement.** `step` is untouched but for its docstring (natural rows weigh one; PR 4's boost stage passes the column). `NetLearner.fit` threads `weight` into `fit_from_scratch` and the held `batch_loss`. The oracle: `values = model.targets[at][legal].astype(np.float64) * np.repeat(model.weight[at].astype(np.float64), width)`. + +- [ ] **Step 4: Run** `uv run pytest tests/deepcfr -q -p no:cacheprovider` → all pass, the oracle pin among them; then `uv run --extra deep pytest tests/deepcfr -q -p no:cacheprovider` for the torch half, then `uv sync`. + +- [ ] **Step 5: Commit** `feat(rung4): the learner seam and the oracle take the weight; the oracle banks Σ t′ · w · r in LCFR's order, so weight one keeps the pin bit for bit` + +### Task 4: the whole suite, a fresh-context review, the PR + +- [ ] `uv run pytest -q -p no:cacheprovider` (about seven minutes; the LCFR 20M run is live, so `nice`). +- [ ] Dispatch a read-only reviewer subagent over `git diff origin/main...` with the plan's Global Constraints and Review Focus; fix what it finds. +- [ ] Push, open the PR against `main` with the house body (what it is, why the weight is where it is, the tests, rulings against the appendix, not in this PR), then watch `claude-review` and answer or fix every finding until none remain. diff --git a/scripts/deepcfr/modal_rung4.py b/scripts/deepcfr/modal_rung4.py index cb671cae..f9f29ae6 100644 --- a/scripts/deepcfr/modal_rung4.py +++ b/scripts/deepcfr/modal_rung4.py @@ -152,7 +152,7 @@ ) TRAINER = {"image": image, "cpu": WORKERS + 1, "memory": MEMORY_GIB * 1024} # The grader holds the compiled game (1.5 GB), a table, a policy memory of -# up to 1.2 GB and Π's fit; eight threads for that fit. +# up to 1.4 GB and Π's fit; eight threads for that fit. GRADER = {"image": image, "cpu": GRADER_CPU, "memory": GRADER_GIB * 1024} def _run_dir(name: str) -> Path: diff --git a/src/drawmaha_solver/deepcfr/AGENTS.md b/src/drawmaha_solver/deepcfr/AGENTS.md index ab52162f..a6fa95d8 100644 --- a/src/drawmaha_solver/deepcfr/AGENTS.md +++ b/src/drawmaha_solver/deepcfr/AGENTS.md @@ -69,9 +69,14 @@ itself is complete; what remains is the paid run and its grades. it must be importable by module name in a fresh process: a module-level function, never a lambda or a closure. The tests prepend the repo root to `sys.path` for exactly this reason. -- A memory row is 24 bytes (`row: int32`, `t: int32`, `target: float32 × 4`); - the run's reservoirs are 50 million rows, 1.2 GB each. A capacity change is - a different run, not a resume. +- A memory row is 28 bytes (`row: int32`, `t: int32`, `target: float32 × 4`, + `weight: float32`); the run's reservoirs are 50 million rows, 1.4 GB each. A + capacity change is a different run, not a resume. The weight is how many + sampled rows the row stands for: 1 for every row a walk offers (what `merge` + stamps when given none), and the targeted-sampling plan's exact rows carry + `(1 − α) · K · arrival`. The refit and the oracle weigh a row by t × weight, + so a memory of weight-one rows is today's run bit for bit, and a memory file + saved before the column existed loads with ones. ## Running it diff --git a/src/drawmaha_solver/deepcfr/__init__.py b/src/drawmaha_solver/deepcfr/__init__.py index 31481960..73399642 100644 --- a/src/drawmaha_solver/deepcfr/__init__.py +++ b/src/drawmaha_solver/deepcfr/__init__.py @@ -13,9 +13,10 @@ - `net`: the MLP, chunked prediction into NumPy, and the save file. - `materialise`: two seats' nets into `cumulative_regret`, Deep CFR's eq. (3) obtained by what is written rather than by changing the ledger. -- `fit`: the from-scratch refit on (row, t, target) arrays. +- `fit`: the from-scratch refit on (row, t, target, weight) arrays. - `reservoir`: the memories those arrays come from — Algorithm R over - (row, t, target) rows, a counter-based stream per phase, an exact resume. + (row, t, target, weight) rows, a counter-based stream per phase, an exact + resume. - `collect`: K lockstep walks for one seat on W spawned processes, returned as the rows a phase offers the memories. - `train`: the loop that calls them in order, iteration after iteration — diff --git a/src/drawmaha_solver/deepcfr/reservoir.py b/src/drawmaha_solver/deepcfr/reservoir.py index 49cf834b..f7315baf 100644 --- a/src/drawmaha_solver/deepcfr/reservoir.py +++ b/src/drawmaha_solver/deepcfr/reservoir.py @@ -1,13 +1,27 @@ -"""The memories: Algorithm R over (row, t, target) rows, with an exact resume. +"""The memories: Algorithm R over (row, t, target, weight) rows, with an exact resume. Deep CFR trains each iteration's net on a MEMORY of what the walks saw: for the advantage memory of seat p, every traverser visit's regret sample r = u − ⟨σ, u⟩ at some table row; for the policy memory, every opponent visit's σ. A row of -either memory is 24 bytes — `row: int32` (which table row), `t: int32` (which -iteration offered it, the refit's weight) and `target: float32 × 4` (r or σ at -the row's legal columns, zero-padded to the widest ledger, 4). Over 2,000 -iterations the walks offer about 280 million advantage rows per seat; the -memory keeps a capacity C of them, 50 million in the run. +either memory is 28 bytes — `row: int32` (which table row), `t: int32` (which +iteration offered it), `target: float32 × 4` (r or σ at the row's legal +columns, zero-padded to the widest ledger, 4) and `weight: float32` (how many +sampled rows this one stands for). The refit weighs a row by t × weight. Over +2,000 iterations the walks offer about 280 million advantage rows per seat; +the memory keeps a capacity C of them, 50 million in the run. + +**The weight.** Every row the walk offers is one sample and weighs 1, which is +what `merge` stamps when no weight is given, so a memory of walk rows alone is +what it was before the column existed. The column is for rows that are NOT one +sample: the targeted-sampling plan's boost stage adds, at a chosen infoset, one +row holding the EXACT regret under the current strategy, weighted +(1 − α) · K · arrival, the number of sampled rows the walk would have put there +in expectation, so the infoset's expected total weight per iteration is +unchanged; and it turns the walk's own rows there down to α. A weight of 0 is a +row the refit must not learn from, kept in the memory so the reservoir sees the +same offers as a run without the boost. The weight rides the same slot as its +row through eviction, save and load, and a file written before the column +existed loads with ones, so a saved run resumes as the run it was. **Which C of the n rows ever offered?** A uniform sample without replacement — every offer resident with probability C/n — so that the refit sees the @@ -34,10 +48,11 @@ fifth word, the ASCII constant "RESV", keeps a reservoir stream from ever equalling a walk's four-word stream. -The three arrays are `np.zeros` at construction, which the kernel pages in -lazily: a 50M-row memory costs 1.2 GB only as it fills, nothing when three of +The four arrays are `np.zeros` at construction, which the kernel pages in +lazily: a 50M-row memory costs 1.4 GB only as it fills, nothing when three of them are built on a laptop. `contents()` is the refit's input — read-only views -of the resident rows, the shape `fit_from_scratch(rows, t, targets, …)` takes. +of the resident rows, the shape `fit_from_scratch(rows, t, targets, weight, …)` +takes. """ from __future__ import annotations @@ -57,9 +72,13 @@ # has four words, so no reservoir stream can equal one. _STREAM_TAG = 0x52455356 -# What a reservoir file holds. `offered` marks a file as a reservoir: no other -# checkpoint in the repo records a count of offers. +# What every reservoir file holds. `offered` marks a file as a reservoir: no +# other checkpoint in the repo records a count of offers. _FILE_KEYS = ("rows", "t", "targets", "offered", "capacity") +# Written by every `save` since the column existed, and read as ones when +# absent: a file without it predates the column, and every row it holds was +# one sample. +_WEIGHT_KEY = "weight" class Memory(IntEnum): """Which memory a stream feeds; the word that tells the two apart in the seed.""" @@ -92,6 +111,7 @@ def __init__(self, capacity: int) -> None: self._rows = np.zeros(self._capacity, dtype=np.int32) self._t = np.zeros(self._capacity, dtype=np.int32) self._targets = np.zeros((self._capacity, TARGET_WIDTH), dtype=np.float32) + self._weight = np.zeros(self._capacity, dtype=np.float32) @property def capacity(self) -> int: @@ -108,14 +128,24 @@ def size(self) -> int: """How many rows are resident: min(n, capacity).""" return min(self._offered, self._capacity) - def merge(self, rows: np.ndarray, targets: np.ndarray, *, t: int, rng) -> None: + def merge( + self, + rows: np.ndarray, + targets: np.ndarray, + *, + t: int, + rng, + weight: np.ndarray | None = None, + ) -> None: """Offer one phase's rows, all stamped `t`, drawing from `rng`. - `rng` needs `Generator.integers(low, high)` with an array `high`: one - draw per late offer. Rows below capacity are kept outright and draw - nothing, so a memory still filling consumes no randomness. + `weight` is one finite, non-negative float per row; left out, every + row weighs 1, the walk's own sample. `rng` needs + `Generator.integers(low, high)` with an array `high`: one draw per + late offer. Rows below capacity are kept outright and draw nothing, + so a memory still filling consumes no randomness. """ - rows, targets = _validate_offer(rows, targets, t) + rows, targets, weight = _validate_offer(rows, targets, t, weight) count = rows.shape[0] capacity = self._capacity # Every offer's global index among all offers ever made. @@ -136,16 +166,17 @@ def merge(self, rows: np.ndarray, targets: np.ndarray, *, t: int, rng) -> None: self._rows[slot[winners]] = rows[winners] self._t[slot[winners]] = t self._targets[slot[winners]] = targets[winners] + self._weight[slot[winners]] = weight[winners] self._offered += count - def contents(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """The resident rows as read-only views: `(rows, t, targets)`, the refit's input. + def contents(self) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: + """The resident rows as read-only views: `(rows, t, targets, weight)`, the refit's input. Live views, not a snapshot: a later `merge` shows through them. The loop takes them after the phase's merge and hands them to the refit. """ size = self.size - views = (self._rows[:size], self._t[:size], self._targets[:size]) + views = (self._rows[:size], self._t[:size], self._targets[:size], self._weight[:size]) for view in views: view.setflags(write=False) return views @@ -159,7 +190,7 @@ def save(self, path: Path) -> None: crash mid-write leaves the previous file whole. """ path = Path(path) - rows, t, targets = self.contents() + rows, t, targets, weight = self.contents() temporary = path.with_name(path.name + ".tmp") with temporary.open("wb") as file: np.savez( @@ -167,6 +198,7 @@ def save(self, path: Path) -> None: rows=rows, t=t, targets=targets, + weight=weight, offered=np.int64(self._offered), capacity=np.int64(self._capacity), ) @@ -179,7 +211,9 @@ def load(cls, path: Path) -> Reservoir: """A memory as `save` left it, ready for the next `merge`. Refuses a file that is not a reservoir (no `offered`) and one whose - resident rows do not match its counts, naming the file either way. + resident rows do not match its counts, naming the file either way. A + file with no `weight` is from before the column existed: its rows + were all plain samples and load weighing one. """ path = Path(path) with np.load(path, allow_pickle=False) as saved: @@ -188,30 +222,40 @@ def load(cls, path: Path) -> Reservoir: raise ValueError(f"{path} is not a reservoir checkpoint (no {missing[0]!r})") capacity, offered = int(saved["capacity"]), int(saved["offered"]) rows, t, targets = saved["rows"], saved["t"], saved["targets"] + weight = saved[_WEIGHT_KEY] if _WEIGHT_KEY in saved.files else None # The constructor would refuse this too, but without the file's name. if capacity < 1: raise ValueError(f"{path}: capacity must be a positive int, got {capacity}") memory = cls(capacity) size = min(offered, capacity) + if weight is None: + weight = np.ones(size, dtype=np.float32) if ( offered < 0 or rows.shape != (size,) or t.shape != (size,) or targets.shape != (size, TARGET_WIDTH) + or weight.shape != (size,) ): raise ValueError( - f"{path} holds {rows.shape[0]} rows, but {offered} offers into a capacity " - f"of {capacity} means {size} resident" + f"{path} holds {rows.shape[0]} rows and {weight.shape[0]} weights, but " + f"{offered} offers into a capacity of {capacity} means {size} resident" ) # The arrays are poured in as they are: a float `rows` would be # truncated into the int32 slots without a word. - expected = (("rows", rows, np.int32), ("t", t, np.int32), ("targets", targets, np.float32)) + expected = ( + ("rows", rows, np.int32), + ("t", t, np.int32), + ("targets", targets, np.float32), + ("weight", weight, np.float32), + ) for name, array, dtype in expected: if array.dtype != dtype: raise ValueError(f"{path}: {name} must be {dtype.__name__}, got {array.dtype}") memory._rows[:size] = rows memory._t[:size] = t memory._targets[:size] = targets + memory._weight[:size] = weight memory._offered = offered return memory @@ -223,10 +267,13 @@ def _is_int(value: object) -> bool: # A bool is an int to Python and never what a caller meant. return isinstance(value, (int, np.integer)) and not isinstance(value, bool) -def _validate_offer(rows: np.ndarray, targets: np.ndarray, t: int) -> tuple[np.ndarray, np.ndarray]: - """The phase's rows as int32 and its targets as float32, or a ValueError naming the fault.""" +def _validate_offer( + rows: np.ndarray, targets: np.ndarray, t: int, weight: np.ndarray | None +) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """The phase's rows as int32, its targets and weights as float32, or a ValueError naming the fault.""" rows = np.asarray(rows) targets = np.asarray(targets) + weight = np.ones(rows.shape[0], dtype=np.float32) if weight is None else np.asarray(weight) # A row is a table row number: a float or a negative is a caller handing # the wrong array. if rows.ndim != 1 or not np.issubdtype(rows.dtype, np.integer): @@ -245,6 +292,13 @@ def _validate_offer(rows: np.ndarray, targets: np.ndarray, t: int) -> tuple[np.n "rows and targets must have the same length, " f"got {rows.shape[0]} and {targets.shape[0]}" ) + if weight.shape != (rows.shape[0],): + raise ValueError(f"weight must be shaped ({rows.shape[0]},), one per row, got {weight.shape}") + # The refit multiplies a row's error by its weight: a negative one would + # pull the net AWAY from the row's target, and a NaN would spread through + # the batch's mean into every row's gradient. + if weight.size and not (np.isfinite(weight).all() and weight.min() >= 0): + raise ValueError("every weight must be finite and non-negative") # Iterations start at 1; the refit weights a row by its t, so 0 would # erase it and a negative would pull against the others. if not _is_int(t) or t < 1: @@ -253,4 +307,8 @@ def _validate_offer(rows: np.ndarray, targets: np.ndarray, t: int) -> tuple[np.n # below the bound, which is here so the stamp can never wrap in silence. if t >= 2**31: raise ValueError(f"t must fit int32, got {t!r}") - return rows.astype(np.int32, copy=False), targets.astype(np.float32, copy=False) + return ( + rows.astype(np.int32, copy=False), + targets.astype(np.float32, copy=False), + weight.astype(np.float32, copy=False), + ) diff --git a/tests/deepcfr/test_reservoir.py b/tests/deepcfr/test_reservoir.py index f7a89541..5d71c17a 100644 --- a/tests/deepcfr/test_reservoir.py +++ b/tests/deepcfr/test_reservoir.py @@ -1,4 +1,4 @@ -"""The reservoir: Algorithm R over (row, t, target) rows, pinned on what the loop relies on. +"""The reservoir: Algorithm R over (row, t, target, weight) rows, pinned on what the loop relies on. Four claims. Below capacity every offer is kept, in order. Above it the memory never grows, counts every offer, and every offer ever made is resident with @@ -6,7 +6,8 @@ A memory saved and loaded merges the next phase exactly as the uninterrupted memory does, because the stream is a function of (seed, t, seat, memory) and the count of offers travels with the checkpoint. And the file format refuses -anything that is not a reservoir. +anything that is not a reservoir, while a file from before the weight column +existed loads with every row weighing one. """ from __future__ import annotations @@ -53,8 +54,9 @@ def filled_memory() -> Reservoir: def test_below_capacity_everything_is_kept_in_order(): memory = Reservoir(capacity=10) memory.merge(np.arange(6), targets(6), t=1, rng=np.random.default_rng(0)) - rows, t, kept = memory.contents() + rows, t, kept, weight = memory.contents() assert memory.offered == 6 and memory.size == 6 + assert weight.tolist() == [1.0] * 6 np.testing.assert_array_equal(rows, np.arange(6)) assert (t == 1).all() np.testing.assert_array_equal(kept, targets(6)) @@ -63,20 +65,23 @@ def test_a_second_merge_below_capacity_appends_after_the_first(): memory = Reservoir(capacity=10) memory.merge(np.arange(4), targets(4), t=1, rng=np.random.default_rng(0)) memory.merge(np.arange(3) + 100, targets(3), t=2, rng=np.random.default_rng(0)) - rows, t, _ = memory.contents() + rows, t, _, _ = memory.contents() assert rows.tolist() == [0, 1, 2, 3, 100, 101, 102] assert t.tolist() == [1, 1, 1, 1, 2, 2, 2] def test_never_exceeds_capacity_and_counts_every_offer(): memory = Reservoir(capacity=10) for t in range(1, 6): - memory.merge(np.arange(7) + 100 * t, targets(7), t=t, rng=np.random.default_rng(t)) + memory.merge( + np.arange(7) + 100 * t, targets(7), t=t, rng=np.random.default_rng(t), weight=np.arange(7) + 0.5 + ) assert memory.offered == 35 and memory.size == 10 - rows, t_col, kept = memory.contents() + rows, t_col, kept, weight = memory.contents() assert set(rows.tolist()) <= {r + 100 * t for t in range(1, 6) for r in range(7)} - # Each row kept its own t and its own target: the three columns move together. + # Each row kept its own t, target and weight: the four columns move together. np.testing.assert_array_equal(t_col, rows // 100) np.testing.assert_array_equal(kept[:, 0], rows % 100) + np.testing.assert_array_equal(weight, rows % 100 + 0.5) def test_a_merge_that_straddles_capacity_fills_then_samples(): memory = Reservoir(capacity=5) @@ -115,11 +120,39 @@ def test_the_later_offer_wins_a_shared_slot(): memory = Reservoir(capacity=2) memory.merge(np.array([0, 1]), targets(2), t=1, rng=np.random.default_rng(0)) memory.merge(np.array([7, 8]), targets(2), t=2, rng=FixedDraws([1, 1])) - rows, t, kept = memory.contents() + rows, t, kept, _ = memory.contents() assert rows.tolist() == [0, 8] assert t.tolist() == [1, 2] assert kept[1, 0] == 1.0 # offer 8 was the second target of its merge +def test_rows_offered_without_a_weight_weigh_one(): + _, _, _, weight = filled_memory().contents() + assert weight.dtype == np.float32 and weight.tolist() == [1.0] * 5 + +def test_a_weight_rides_its_row_through_eviction(): + # The straddle case above, with a distinct weight per offer: slot 0 is + # overwritten by offer 12, whose weight is 4.0, and the dropped offer 13 + # leaves no weight behind. + memory = Reservoir(capacity=5) + memory.merge(np.arange(3), targets(3), t=1, rng=np.random.default_rng(0), weight=np.array([0.5, 1.0, 1.5])) + memory.merge(np.arange(4) + 10, targets(4), t=2, rng=FixedDraws([0, 7]), weight=np.array([2.0, 3.0, 4.0, 5.0])) + rows, _, _, weight = memory.contents() + assert rows.tolist() == [12, 1, 2, 10, 11] + assert weight.tolist() == [4.0, 1.0, 1.5, 2.0, 3.0] + +def test_merge_refuses_bad_weights(): + memory = Reservoir(capacity=10) + rng = np.random.default_rng(0) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, -0.5])) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, np.nan])) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 1.0, 1.0])) + with pytest.raises(ValueError, match="weight"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.ones((2, 1))) + assert memory.offered == 0 # a refused offer changes nothing + def test_an_empty_merge_changes_nothing(): memory = filled_memory() before = [a.copy() for a in memory.contents()] @@ -204,13 +237,17 @@ def test_merge_refuses_bad_rows(): Reservoir(capacity=True) def test_contents_are_read_only_and_feed_the_fit(): - rows, t, tg = filled_memory().contents() + rows, t, tg, weight = filled_memory().contents() assert rows.dtype == np.int32 and t.dtype == np.int32 and tg.dtype == np.float32 + assert weight.dtype == np.float32 assert rows.shape == (5,) and t.shape == (5,) and tg.shape == (5, TARGET_WIDTH) + assert weight.shape == (5,) with pytest.raises(ValueError): rows[0] = 1 with pytest.raises(ValueError): tg[0, 0] = 1.0 + with pytest.raises(ValueError): + weight[0] = 2.0 # --------------------------------------------------------------------------- # The file @@ -270,6 +307,43 @@ def test_load_refuses_arrays_of_the_wrong_type(tmp_path): with pytest.raises(ValueError, match="rows must be int32"): Reservoir.load(broken) +def test_a_file_written_before_the_weight_existed_loads_with_ones(tmp_path): + # Every memory saved before PR 2 of the targeted-sampling plan has no + # `weight` key; its rows were all plain samples, so they weigh one. + old = tmp_path / "advantage-p0-t00007.npz" + np.savez( + old, + rows=np.arange(5, dtype=np.int32), + t=np.full(5, 7, np.int32), + targets=targets(5), + offered=np.int64(5), + capacity=np.int64(8), + ) + loaded = Reservoir.load(old) + rows, t, kept, weight = loaded.contents() + assert weight.dtype == np.float32 and weight.tolist() == [1.0] * 5 + loaded.merge(np.arange(3) + 50, targets(3), t=8, rng=np.random.default_rng(0), weight=np.array([2.0, 2.0, 2.0])) + assert loaded.contents()[3].tolist() == [1.0] * 5 + [2.0] * 3 + +def test_load_refuses_a_weight_of_the_wrong_length_or_type(tmp_path): + def write(name, weight): + path = tmp_path / name + np.savez( + path, + rows=np.arange(3, dtype=np.int32), + t=np.ones(3, np.int32), + targets=targets(3), + weight=weight, + offered=np.int64(3), + capacity=np.int64(8), + ) + return path + + with pytest.raises(ValueError, match="short.npz"): + Reservoir.load(write("short.npz", np.ones(2, np.float32))) + with pytest.raises(ValueError, match="weight must be float32"): + Reservoir.load(write("double.npz", np.ones(3, np.float64))) + def test_a_loaded_memory_can_keep_filling_to_its_capacity(tmp_path): memory = filled_memory() memory.save(tmp_path / "m.npz") From a8e5a43c5979f433d2c68455d85a6ee19f912048 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 17:35:47 -0500 Subject: [PATCH 08/16] =?UTF-8?q?feat(rung4):=20the=20refit=20weighs=20a?= =?UTF-8?q?=20row=20by=20t=20=C3=97=20weight;=20weight=20one=20is=20today'?= =?UTF-8?q?s=20loss=20bit=20for=20bit=20and=20a=20row=20of=20weight=20zero?= =?UTF-8?q?=20is=20inert?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second of PR 2's three commits. _weighted_loss scales each row's squared error by (t · w) / mean(t · w): with w ≡ 1 that is t / mean(t) in the same bits, since t · 1.0 is t in float32. fit_from_scratch and batch_loss take the weight as a fourth positional array, None meaning ones, and refuse a weight that is not 1-D, not the memory's length, non-finite, negative, or all zero (nothing to fit, as an empty memory is). A batch whose every row weighs 0 divides by a clamped mean and has loss 0, not NaN. fit_policy takes the column positionally so the policy memory's contents() unpack straight in; its weights are all one. --- scripts/deepcfr/pi_loss_sweep.py | 2 +- src/drawmaha_solver/deepcfr/fit.py | 107 ++++++++++++++++++-------- src/drawmaha_solver/deepcfr/policy.py | 11 ++- tests/deepcfr/test_fit.py | 60 +++++++++++++++ 4 files changed, 146 insertions(+), 34 deletions(-) diff --git a/scripts/deepcfr/pi_loss_sweep.py b/scripts/deepcfr/pi_loss_sweep.py index b59aef1b..cf35b090 100644 --- a/scripts/deepcfr/pi_loss_sweep.py +++ b/scripts/deepcfr/pi_loss_sweep.py @@ -222,7 +222,7 @@ def main() -> None: continue mark = mark_of(arm) if loaded is None or loaded[0] != mark: - rows, t, targets = Reservoir.load(run_dir / "marks" / f"t{mark:05d}" / f"policy-t{mark:05d}.npz").contents() + rows, t, targets, _ = Reservoir.load(run_dir / "marks" / f"t{mark:05d}" / f"policy-t{mark:05d}.npz").contents() exact = load_strategy(run_dir / "frozen" / f"t{mark:05d}-exact.npz") loaded = (mark,) print(f"mark t={mark}: memory {len(rows):,} rows", flush=True) diff --git a/src/drawmaha_solver/deepcfr/fit.py b/src/drawmaha_solver/deepcfr/fit.py index 42f07949..9bb4546e 100644 --- a/src/drawmaha_solver/deepcfr/fit.py +++ b/src/drawmaha_solver/deepcfr/fit.py @@ -1,26 +1,35 @@ -"""The from-scratch refit: a fresh net fitted to (row, t, target) rows. +"""The from-scratch refit: a fresh net fitted to (row, t, target, weight) rows. Deep CFR does not update a net between iterations; it throws the net away -and fits a new one to everything the memory holds. The memory is three +and fits a new one to everything the memory holds. The memory is four arrays the same length: `rows`, the table row each sample was taken at (the net never sees the row number — `Layout.features` turns it into the 136 floats the player could see); `t`, the iteration the sample was taken -on; and `targets`, what the net should say at that row's legal columns, -`MAX_WIDTH` wide with padding past the row's width. Nothing here knows -what a target IS — a sampled regret, an action probability — so the same -fit serves the regret nets and the policy net. +on; `targets`, what the net should say at that row's legal columns, +`MAX_WIDTH` wide with padding past the row's width; and `weight`, how many +sampled rows the row stands for, 1 for a walk's own sample (see +`reservoir.py` for the rows that are not one). Nothing here knows what a +target IS — a sampled regret, an action probability — so the same fit +serves the regret nets and the policy net. **The loss.** Per row, the squared error summed over its legal columns, read off the head through `features.legal_gather()`; the padding is masked out BEFORE the subtraction, so a padded column holding NaN is never looked -at and never leaks into a gradient. Each row is weighted by its iteration -t, divided by the batch's mean t so the loss keeps one scale as the run +at and never leaks into a gradient. Each row is weighted by t × weight — +its iteration times how many samples it stands for — divided by the +batch's mean of t × weight so the loss keeps one scale as the run lengthens, and the batch loss is the mean over rows: a 2-wide and a 4-wide -spot each count as one row. The weight is Deep CFR's (Brown, Lerer, Gross & +spot each count as one row. The t is Deep CFR's (Brown, Lerer, Gross & Sandholm 2019), borrowed from Linear CFR (Brown & Sandholm 2019): the early iterations' strategies are far from equilibrium, so the regrets sampled against them say the least about the game, and a weight that -grows with t lets the later samples outvote them. +grows with t lets the later samples outvote them. The weight multiplies t +rather than the target, so it changes how much a row counts and never +what the row says; at weight 1 throughout it is t / mean(t) bit for bit +(t · 1.0 is t in float32), and a row of weight 0 contributes nothing. A +batch whose every row weighs 0 has a mean of 0 to divide by; its loss is +0 rather than 0/0, so a tiny memory under the boost cannot end a fit as +"diverged" on one unlucky draw. **The optimiser.** Adam at the spec's learning rate, the gradient's norm clipped to the spec's bound before each step so one wild sample cannot @@ -49,7 +58,7 @@ feature halves and the legal gather go to the device once per fit and each batch's features are gathered there, so no step goes back through numpy. That copy holds the whole memory a second time for the fit's duration -(rows, t, and `MAX_WIDTH` float32 targets per row), so peak memory is +(rows, t, weight, and `MAX_WIDTH` float32 targets per row), so peak memory is about twice the memory's; unified memory makes the copy cheap, not free. Only the order of the arithmetic changes, so it is the same rule but not the same net. Adam divides by √v, so a near-zero gradient component turns @@ -116,34 +125,48 @@ def __post_init__(self) -> None: # The loss # --------------------------------------------------------------------------- -def batch_loss(net: Net, rows: np.ndarray, t: np.ndarray, targets: np.ndarray) -> torch.Tensor: +def batch_loss( + net: Net, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray | None = None, +) -> torch.Tensor: """One batch's loss, a scalar tensor: what every step minimises. Public so a test and PR 4's logging read the same number the optimiser - does. Builds the batch's features on the fly from `layout()`. Assumes - what `fit_from_scratch` checks: every t is >= 1, so the batch's mean t - the weights are divided by is positive. + does. Builds the batch's features on the fly from `layout()`. `weight` + left out is a batch of plain samples, every row weighing 1. Assumes + what `fit_from_scratch` checks: every t is >= 1 and every weight is + finite and non-negative. """ built = layout() point, _ = built.locate_rows(rows) gather = torch.from_numpy(legal_gather()[point].astype(np.int64)) + if weight is None: + weight = np.ones(len(rows), dtype=np.float32) return _weighted_loss( net(torch.from_numpy(built.features(rows))), gather, torch.from_numpy(np.asarray(targets, dtype=np.float32)), torch.from_numpy(np.asarray(t, dtype=np.float32)), + torch.from_numpy(np.asarray(weight, dtype=np.float32)), ) def _weighted_loss( - outputs: torch.Tensor, gather: torch.Tensor, targets: torch.Tensor, t: torch.Tensor + outputs: torch.Tensor, + gather: torch.Tensor, + targets: torch.Tensor, + t: torch.Tensor, + weight: torch.Tensor, ) -> torch.Tensor: """The loss from the net's outputs on: the one copy both the cpu and the device fit call. `outputs` (b, head) the net's raw head, `gather` (b, MAX_WIDTH) each - row's ledger columns with `NO_ACTION` padding, `targets` (b, MAX_WIDTH) - and `t` (b,) float32, all on one device. Everything before this — the - features, the gather's rows — differs between the paths only in where - it is looked up, never in what it is. + row's ledger columns with `NO_ACTION` padding, `targets` (b, MAX_WIDTH), + `t` (b,) and `weight` (b,) float32, all on one device. Everything + before this — the features, the gather's rows — differs between the + paths only in where it is looked up, never in what it is. """ legal = gather != NO_ACTION predicted = outputs.gather(1, gather.clamp(min=0)) @@ -151,7 +174,11 @@ def _weighted_loss( # survive `where` in the backward pass, where 0 * NaN is NaN. target = torch.where(legal, targets, 0.0) error = torch.where(legal, (predicted - target) ** 2, 0.0).sum(dim=1) - weight = t / t.mean() + scale = t * weight + # A batch of weight-0 rows alone has a mean of 0; clamped, its loss is 0 + # and not NaN. Any positive mean is far above `tiny`, so the clamp + # returns it unchanged and the division is the plain one. + weight = scale / scale.mean().clamp(min=torch.finfo(scale.dtype).tiny) return (weight * error).mean() # --------------------------------------------------------------------------- @@ -162,6 +189,7 @@ def fit_from_scratch( rows: np.ndarray, t: np.ndarray, targets: np.ndarray, + weight: np.ndarray | None = None, *, seed: tuple[int, ...], config: FitConfig = FitConfig(), @@ -171,13 +199,16 @@ def fit_from_scratch( `rows` (n,) int table rows, `t` (n,) iteration numbers >= 1 (an integer-valued float is accepted: t enters only as a float32 weight), `targets` (n, MAX_WIDTH) float in ledger-column order with padding - ignored. + ignored, `weight` (n,) finite and >= 0, or None for a memory of plain + samples, every row weighing 1. Refused by name before anything is built: - - a `rows` or `t` that is not 1-D, or non-integer rows; + - a `rows`, `t` or `weight` that is not 1-D, or non-integer rows; - mismatched lengths, or an empty memory; - a non-finite t, or one below 1; + - a non-finite or negative weight, or weights with no positive entry + (nothing to fit, as an empty memory is); - targets not `MAX_WIDTH` wide, or non-finite at a legal column; - a row outside the table. @@ -186,16 +217,20 @@ def fit_from_scratch( rows = np.asarray(rows) t = np.asarray(t) targets = np.asarray(targets, dtype=np.float32) + weight = np.ones(len(rows), dtype=np.float32) if weight is None else np.asarray(weight, dtype=np.float32) # A (n, 1) t would broadcast against the (n,) error into an (n, n) loss # and train on the wrong number without a word, so the shapes are # refused by name, not left to broadcasting. - if rows.ndim != 1 or t.ndim != 1: - raise ValueError(f"rows and t must be 1-D, got shapes {rows.shape} and {t.shape}") + if rows.ndim != 1 or t.ndim != 1 or weight.ndim != 1: + raise ValueError( + f"rows, t and weight must be 1-D, got shapes {rows.shape}, {t.shape} and {weight.shape}" + ) if not np.issubdtype(rows.dtype, np.integer): raise ValueError(f"rows must be integer row numbers, got dtype {rows.dtype}") - if not len(rows) == len(t) == len(targets): + if not len(rows) == len(t) == len(targets) == len(weight): raise ValueError( - f"rows, t and targets must have the same length, got {len(rows)}, {len(t)}, {len(targets)}" + "rows, t, targets and weight must have the same length, " + f"got {len(rows)}, {len(t)}, {len(targets)}, {len(weight)}" ) if len(rows) == 0: raise ValueError("the memory is empty: nothing to fit") @@ -205,6 +240,12 @@ def fit_from_scratch( raise ValueError("every t must be finite") if t.min() < 1: raise ValueError(f"every t must be an iteration number >= 1, got a minimum of {t.min()}") + if not np.isfinite(weight).all() or weight.min() < 0: + raise ValueError("every weight must be finite and non-negative") + # Every row inert: the batches would all be the weightless case the + # loss guards, and 4,000 steps of zero gradient would "fit" nothing. + if not (weight > 0).any(): + raise ValueError("no row carries weight: nothing to fit") rows = rows.astype(np.int64) point, _ = layout().locate_rows(rows) # raises IndexError for a row outside the table # Only the legal columns are checked: the padding may hold anything. @@ -215,7 +256,7 @@ def fit_from_scratch( # config can never quietly run the cpu loop; the device loop checks only # mps's availability, so a new name must add its own check there. if config.device != "cpu": - return _fit_on_device(rows, t, targets, folded, config, device=config.device) + return _fit_on_device(rows, t, targets, weight, folded, config, device=config.device) threads_before = torch.get_num_threads() torch.set_num_threads(config.threads) try: @@ -229,7 +270,7 @@ def fit_from_scratch( # (the cpu-parity test in test_fit.py is the guard). picked = torch.randint(len(rows), (config.batch,), generator=draw).numpy() optimiser.zero_grad() - batch_loss(net, rows[picked], t[picked], targets[picked]).backward() + batch_loss(net, rows[picked], t[picked], targets[picked], weight[picked]).backward() clip_grad_norm_(net.parameters(), config.clip) optimiser.step() finally: @@ -245,6 +286,7 @@ def _fit_on_device( rows: np.ndarray, t: np.ndarray, targets: np.ndarray, + weight: np.ndarray, folded: int, config: FitConfig, *, @@ -274,6 +316,7 @@ def _fit_on_device( gather_all = torch.from_numpy(legal_gather().astype(np.int64)).to(device) memory_rows = torch.from_numpy(rows).to(device) memory_t = torch.from_numpy(np.asarray(t, dtype=np.float32)).to(device) + memory_weight = torch.from_numpy(np.asarray(weight, dtype=np.float32)).to(device) # `from_numpy` warns on a read-only array (a memory loaded from a file # can be one); `require` copies only in that case. memory_targets = torch.from_numpy(np.require(targets, requirements="W")).to(device) @@ -292,7 +335,11 @@ def _fit_on_device( position = batch_rows - row_offset[point] features = torch.cat([private[private_offset[point] + position], public[point]], dim=1) loss = _weighted_loss( - net(features), gather_all[point], memory_targets[picked], memory_t[picked] + net(features), + gather_all[point], + memory_targets[picked], + memory_t[picked], + memory_weight[picked], ) optimiser.zero_grad() loss.backward() diff --git a/src/drawmaha_solver/deepcfr/policy.py b/src/drawmaha_solver/deepcfr/policy.py index fc8e7ed0..24a907b9 100644 --- a/src/drawmaha_solver/deepcfr/policy.py +++ b/src/drawmaha_solver/deepcfr/policy.py @@ -23,7 +23,9 @@ **The fit.** `fit_from_scratch` on the memory's `contents()`; the targets are σ rows zero-padded to `TARGET_WIDTH`, and `batch_loss` masks the -padding before the subtraction, as it does for advantage rows. Nothing in +padding before the subtraction, as it does for advantage rows. The policy +memory's weights are all 1 — the opponent's walks are the only thing that +offers σ rows — so the column is carried and changes nothing here. Nothing in the fit knows the target is a probability, so a regret memory handed by mistake would fit in silence; `fit_policy` refuses one by checking that every row is a distribution at its legal columns. A fresh net every time, @@ -84,13 +86,16 @@ def fit_policy( rows: np.ndarray, t: np.ndarray, targets: np.ndarray, + weight: np.ndarray | None = None, *, seed: int, iteration: int, config: FitConfig | None = None, ) -> Net: - """A fresh net fit to the policy memory's `(rows, t, targets)` at `iteration`, under `policy_seed`. + """A fresh net fit to the policy memory's `(rows, t, targets, weight)` at `iteration`, under `policy_seed`. + `weight` is the memory's fourth column, positional so the memory's + `contents()` unpacks straight in; None is a memory of plain samples. `config` defaults to `FitConfig()`, the spec's refit. Refuses, before anything is built, a targets row that is not a distribution at its legal columns — a regret memory handed in place of the policy memory. @@ -100,7 +105,7 @@ def fit_policy( targets = np.asarray(targets, dtype=np.float32) _require_distributions(np.asarray(rows), targets) return fit_from_scratch( - rows, t, targets, seed=policy_seed(seed, iteration), config=config or FitConfig() + rows, t, targets, weight, seed=policy_seed(seed, iteration), config=config or FitConfig() ) def _require_distributions(rows: np.ndarray, targets: np.ndarray) -> None: diff --git a/tests/deepcfr/test_fit.py b/tests/deepcfr/test_fit.py index 431cb957..031feb64 100644 --- a/tests/deepcfr/test_fit.py +++ b/tests/deepcfr/test_fit.py @@ -101,6 +101,65 @@ def test_later_iterations_weigh_more(): ) np.testing.assert_allclose(legal_prediction(net, row), 0.5, atol=0.05) +def test_weight_one_is_the_unweighted_fit_bit_for_bit(): + # t · 1.0 is t exactly in float32, so the loss, the gradients and the + # weights are the same bits as a fit told nothing about weights. + rows, t, targets = real_rows(300, seed=4) + plain = fit_from_scratch(rows, t, targets, seed=(0, 1, 0), config=SMALL) + ones = fit_from_scratch(rows, t, targets, np.ones(300, np.float32), seed=(0, 1, 0), config=SMALL) + for a, b in zip(plain.parameters(), ones.parameters(), strict=True): + assert torch.equal(a, b) + +def test_a_row_of_weight_zero_is_inert(): + # One row twice at t = 1: +1 at weight 0 and −1 at weight 1. The first + # row must leave no trace, so the fit lands on −1, not on the mean 0. + (row,), _, _ = real_rows(1, seed=3) + rows = np.array([row, row]) + t = np.array([1, 1]) + targets = np.array([[1, 1, 1, 1], [-1, -1, -1, -1]], dtype=np.float32) + net = fit_from_scratch( + rows, t, targets, np.array([0.0, 1.0]), seed=(0, 1, 0), config=FitConfig(steps=1_500, batch=64, threads=1) + ) + np.testing.assert_allclose(legal_prediction(net, row), -1.0, atol=0.05) + +def test_the_loss_weighs_a_row_by_t_times_weight(): + # The same two rows as `test_later_iterations_weigh_more`, +1 at t = 3 and + # −1 at t = 1, with weights (1, 3): t · w is 3 for both, so the weighted + # least-squares answer moves from +0.5 to the plain mean, 0. + (row,), _, _ = real_rows(1, seed=3) + rows = np.array([row, row]) + t = np.array([3, 1]) + targets = np.array([[1, 1, 1, 1], [-1, -1, -1, -1]], dtype=np.float32) + net = fit_from_scratch( + rows, t, targets, np.array([1.0, 3.0]), seed=(0, 1, 0), config=FitConfig(steps=1_500, batch=64, threads=1) + ) + np.testing.assert_allclose(legal_prediction(net, row), 0.0, atol=0.05) + +def test_a_batch_of_weightless_rows_has_a_finite_loss(): + # Every row of the batch at weight 0 (a tiny memory under the boost's α = 0): + # the normalising mean is 0, and 0/0 would make the loss NaN and the fit + # "diverged". The loss is 0 instead, and its gradient finite. + rows, t, targets = real_rows(16, seed=8) + torch.manual_seed(0) + net = Net() + loss = batch_loss(net, rows, t, targets, np.zeros(16, np.float32)) + assert loss.item() == 0.0 + loss.backward() + assert all(torch.isfinite(p.grad).all() for p in net.parameters()) + +def test_fit_refuses_bad_weights(): + rows, t, targets = real_rows(10, seed=3) + with pytest.raises(ValueError, match="weight"): + fit_from_scratch(rows, t, targets, np.r_[np.ones(9), -1.0], seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="weight"): + fit_from_scratch(rows, t, targets, np.r_[np.ones(9), np.nan], seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="length"): + fit_from_scratch(rows, t, targets, np.ones(9), seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="1-D"): + fit_from_scratch(rows, t, targets, np.ones((10, 1)), seed=(0,), config=SMALL) + with pytest.raises(ValueError, match="nothing to fit"): + fit_from_scratch(rows, t, targets, np.zeros(10), seed=(0,), config=SMALL) + def test_loss_is_over_the_legal_columns_only(): # A 2-wide betting row: its loss is the sum over its two columns, so a # target met exactly at both leaves zero whatever the padding says. @@ -267,6 +326,7 @@ def test_the_device_loop_on_the_cpu_is_the_cpu_fit_bit_for_bit(): rows.astype(np.int64), t, targets.astype(np.float32), + np.ones(len(rows), np.float32), fit_module._fold_seed((0, 2, 1)), SMALL, device="cpu", From 6cff997bff30189aee46a143d3c4b6e9c779c60a Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 17:40:42 -0500 Subject: [PATCH 09/16] =?UTF-8?q?feat(rung4):=20the=20learner=20seam=20and?= =?UTF-8?q?=20the=20oracle=20take=20the=20weight;=20the=20oracle=20banks?= =?UTF-8?q?=20=CE=A3=20t=E2=80=B2=20=C2=B7=20w=20=C2=B7=20r=20in=20LCFR's?= =?UTF-8?q?=20order,=20so=20weight=20one=20keeps=20the=20pin=20bit=20for?= =?UTF-8?q?=20bit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Last of PR 2's three commits. Learner.fit takes the memory's four arrays, the weight positional with no default so no learner can drop it by accident; step is unchanged, the four-tuple from contents() flowing through. NetLearner hands the weight to fit_from_scratch and the held batch's loss, after the pot scaling and the regret rules act on the target: the weight says how loudly a row counts, never what it says. The oracle multiplies each sample by its weight in float64 before the per-t′ np.add.at sum, where × 1.0 is exact, so the LCFR pin passes unchanged; a weight of 2 banks what the row offered twice would, a weight of 0 nothing. Oracle model files carry the weight and load as ones without it. --- src/drawmaha_solver/deepcfr/AGENTS.md | 7 +- src/drawmaha_solver/deepcfr/train.py | 105 +++++++++++++++++++------- tests/deepcfr/test_train.py | 42 +++++++++-- tests/deepcfr/test_train_torch.py | 24 +++--- 4 files changed, 133 insertions(+), 45 deletions(-) diff --git a/src/drawmaha_solver/deepcfr/AGENTS.md b/src/drawmaha_solver/deepcfr/AGENTS.md index a6fa95d8..4f0673f1 100644 --- a/src/drawmaha_solver/deepcfr/AGENTS.md +++ b/src/drawmaha_solver/deepcfr/AGENTS.md @@ -22,9 +22,10 @@ itself is complete; what remains is the paid run and its grades. game sit behind `MINIDRAWMAHA_FULL_DEEP=1`. - **The oracle pin is the correctness gate.** With `OracleLearner` the loop must reproduce rung 3's LCFR table bit for bit over three iterations. A - change to the loop's order, to the float32 cast in `collect.rows_of`, or to - `regret_rules.bank_rows` breaks that pin. If it breaks, the loop is wrong, - not the test. + change to the loop's order, to the float32 cast in `collect.rows_of`, to + `regret_rules.bank_rows`, or to where the row weight multiplies (it scales + each sample in float64 before LCFR's per-t′ sum, so weight one is exact) + breaks that pin. If it breaks, the loop is wrong, not the test. - **A learner's `write` assigns into `table.cumulative_regret` and never rebinds it.** The collector's workers read that array through a shared block; a rebound array leaves them walking stale regrets. diff --git a/src/drawmaha_solver/deepcfr/train.py b/src/drawmaha_solver/deepcfr/train.py index 9aba3170..891ada27 100644 --- a/src/drawmaha_solver/deepcfr/train.py +++ b/src/drawmaha_solver/deepcfr/train.py @@ -12,7 +12,7 @@ rows = collector.collect(seat=p, t=t, walks=K) advantage[p].merge(rows.advantage_rows, rows.advantage_targets, t=t, rng=stream(ADVANTAGE)) policy.merge(rows.policy_rows, rows.policy_targets, t=t, rng=stream(POLICY)) - nets[p] = learner.fit(*advantage[p].contents(), seed=(seed, t, p)).model + nets[p] = learner.fit(*advantage[p].contents(), seed=(seed, t, p)).model # rows, t, targets, weight — plus everything a 15-hour run needs around it: the resume state, grid-mark saves, a progress line per iteration, and a graceful stop on a signal or a @@ -33,8 +33,11 @@ imports `fit`, `materialise` and `net` inside those methods. That is what lets the whole-loop test run in the default suite with no torch: the `OracleLearner` below is a "perfect net" that returns, at every row, the -t′-weighted sum of every regret sample the memory holds there, and with it -Deep CFR collapses into rung 3's tabular LCFR. The one thing the loop asks +sum over every regret sample the memory holds there of t′ × weight × r, and +with it Deep CFR collapses into rung 3's tabular LCFR. The weight is the +memory's fourth column (`reservoir.py`): 1 for every row a walk offers, so +until the targeted-sampling plan's boost stage injects rows that stand for +more or fewer samples than one, it multiplies nothing. The one thing the loop asks of a `write` is that it assigns INTO `table.cumulative_regret` and never rebinds it: the collector's workers read that array through a shared block, and a rebound array would leave them walking stale regrets. @@ -53,7 +56,10 @@ reordered row) still breaks that — and to float32 precision against the untouched `train_lockstep`. The oracle repeats LCFR's float operations in LCFR's order (per t′ in increasing order, `R[cols] += t′ · totals`) because -a single Σ t′·r over all rows would round differently from t = 2 on. +a single Σ t′·r over all rows would round differently from t = 2 on. The +weight multiplies each sample before that sum, in float64, where × 1.0 is +exact: a memory of weight-one rows banks the same bits as before the column +existed, which is what keeps the pin a pin. **The resume state** is three memories, the two latest nets, the file of every observer that carries state (PR 5's `ExactAverage`: an observer with @@ -311,14 +317,23 @@ class Fitted: class Learner(Protocol): """The one seam between the loop and the model. - `fit` takes a memory's `contents()` and the fit's seed `(seed, t, seat)`; - `write` plays `models[seat]` for both seats by assigning into - `table.cumulative_regret`; `save` and `load` round-trip one model - through one file, for the resume state and the kept nets. + `fit` takes a memory's `contents()` — rows, t, targets and the weight + of each row, how many samples it stands for — and the fit's seed + `(seed, t, seat)`; `write` plays `models[seat]` for both seats by + assigning into `table.cumulative_regret`; `save` and `load` round-trip + one model through one file, for the resume state and the kept nets. + The weight is positional and has no default here, so no learner can + take the memory and drop its weights by accident. """ def fit( - self, rows: np.ndarray, t: np.ndarray, targets: np.ndarray, *, seed: tuple[int, int, int] + self, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray, + *, + seed: tuple[int, int, int], ) -> Fitted: ... def write(self, table: PackedTable, models: Sequence[object | None]) -> None: ... @@ -388,7 +403,10 @@ def step( `collector` is a `Collector` (or anything with its `table` and `collect`); the iteration is `run.t + 1`, and `run.t` advances once both phases are done. A raise mid-way leaves `run` half-advanced (seat 0's - net and memories updated, `t` not), which `run_deep` never saves. + net and memories updated, `t` not), which `run_deep` never saves. The + walks' rows are merged without a weight and so weigh one each; the + targeted-sampling plan's boost stage is what will offer rows that weigh + otherwise, after the natural rows so these keep their stream positions. """ t = run.t + 1 config = run.config @@ -1037,7 +1055,13 @@ def __init__(self, config: DeepConfig) -> None: self._fit_config = None def fit( - self, rows: np.ndarray, t: np.ndarray, targets: np.ndarray, *, seed: tuple[int, int, int] + self, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray, + *, + seed: tuple[int, int, int], ) -> Fitted: import torch @@ -1047,12 +1071,15 @@ def fit( if self._config.rule != "linear": targets = rule_targets(targets, t, now=seed[1], rule=self._config.rule) t = np.ones(len(t), dtype=np.float32) + # The weight is a sample weight, how much the row counts, and never + # touches the target: the pot scaling and the rules act on what the + # row says, the weight on how loudly, and the two compose. targets = pot_scaled(rows, targets, power=self._config.pot_power) - net = fit_from_scratch(rows, t, targets, seed=seed, config=config) + net = fit_from_scratch(rows, t, targets, weight, seed=seed, config=config) draw = torch.Generator().manual_seed(_fold_seed((*seed, _HELD_TAG))) held = torch.randint(len(rows), (config.batch,), generator=draw).numpy() with torch.inference_mode(): - loss = float(batch_loss(net, rows[held], t[held], targets[held]).item()) + loss = float(batch_loss(net, rows[held], t[held], targets[held], weight[held]).item()) return Fitted(net, loss) def write(self, table: PackedTable, models: Sequence[object | None]) -> None: @@ -1117,19 +1144,24 @@ class OracleModel: rows: np.ndarray # int32 (n,) t: np.ndarray # int32 (n,) targets: np.ndarray # float32 (n, TARGET_WIDTH) + weight: np.ndarray # float32 (n,) class OracleLearner: """The learner that makes the loop LCFR: its model is the memory, its write is LCFR's bank. - `fit` copies the memory. `write` puts at every row of each seat the - t′-weighted sum of the samples its model holds there, computed as LCFR - computes it — per t′ in increasing order, the samples of that t′ summed - per column in memory order with `np.add.at`, then `R[cols] += t′ · - totals` — so the bits match `bank_rows` through any number of - iterations, not only the first. The two models are summed one after - the other, which is only LCFR's order because a row belongs to one - acting seat and so appears in one model. A row no sample names stays - exactly 0.0. NumPy only, so gate 1 runs without torch. + `fit` copies the memory. `write` puts at every row of each seat the sum + of t′ × weight × sample over the samples its model holds there, computed + as LCFR computes it — per t′ in increasing order, the samples of that + t′ each multiplied by their weight in float64 and summed per column in + memory order with `np.add.at`, then `R[cols] += t′ · totals` — so the + bits match `bank_rows` through any number of iterations, not only the + first: × 1.0 is exact, so a memory of weight-one rows banks what it + banked before the column existed. A row of weight 2 banks what the same + row offered twice would (r + r = 2r exactly), and a row of weight 0 + banks nothing. The two models are summed one after the other, which is + only LCFR's order because a row belongs to one acting seat and so + appears in one model. A row no sample names stays exactly 0.0. NumPy + only, so gate 1 runs without torch. """ def __init__(self) -> None: @@ -1137,9 +1169,17 @@ def __init__(self) -> None: self._starts: dict[int, tuple[np.ndarray, np.ndarray]] = {} def fit( - self, rows: np.ndarray, t: np.ndarray, targets: np.ndarray, *, seed: tuple[int, int, int] + self, + rows: np.ndarray, + t: np.ndarray, + targets: np.ndarray, + weight: np.ndarray, + *, + seed: tuple[int, int, int], ) -> Fitted: - return Fitted(OracleModel(np.array(rows), np.array(t), np.array(targets)), None) + return Fitted( + OracleModel(np.array(rows), np.array(t), np.array(targets), np.array(weight)), None + ) def write(self, table: PackedTable, models: Sequence[OracleModel | None]) -> None: starts, widths = self._layout(table) @@ -1153,7 +1193,11 @@ def write(self, table: PackedTable, models: Sequence[OracleModel | None]) -> Non width = widths[rows] columns = _columns(starts[rows], width) legal = np.arange(TARGET_WIDTH)[None, :] < width[:, None] - values = model.targets[at][legal].astype(np.float64) + # `legal` reads row by row, so each row's weight repeats once + # per legal column, in the same order as its values. + values = model.targets[at][legal].astype(np.float64) * np.repeat( + model.weight[at].astype(np.float64), width + ) touched, _, inverse = np.unique(columns, return_index=True, return_inverse=True) totals = np.zeros(touched.size) np.add.at(totals, inverse, values) @@ -1164,11 +1208,18 @@ def write(self, table: PackedTable, models: Sequence[OracleModel | None]) -> Non def save(self, model: OracleModel, path: Path) -> None: # Through an open file, so NumPy does not append `.npz` to the name. with Path(path).open("wb") as file: - np.savez(file, rows=model.rows, t=model.t, targets=model.targets) + np.savez(file, rows=model.rows, t=model.t, targets=model.targets, weight=model.weight) def load(self, path: Path) -> OracleModel: + # A model saved before the weight column existed held plain samples, + # one each, as a reservoir file without the key does. with np.load(Path(path), allow_pickle=False) as saved: - return OracleModel(saved["rows"], saved["t"], saved["targets"]) + weight = ( + saved["weight"] + if "weight" in saved.files + else np.ones(saved["rows"].shape[0], dtype=np.float32) + ) + return OracleModel(saved["rows"], saved["t"], saved["targets"], weight) def _layout(self, table: PackedTable) -> tuple[np.ndarray, np.ndarray]: if id(table) not in self._starts: diff --git a/tests/deepcfr/test_train.py b/tests/deepcfr/test_train.py index e8c51c6d..8047deb7 100644 --- a/tests/deepcfr/test_train.py +++ b/tests/deepcfr/test_train.py @@ -24,7 +24,7 @@ import pytest from drawmaha_solver.deepcfr.collect import Collector -from drawmaha_solver.deepcfr.reservoir import Memory, Reservoir, reservoir_stream +from drawmaha_solver.deepcfr.reservoir import TARGET_WIDTH, Memory, Reservoir, reservoir_stream import drawmaha_solver.deepcfr.train as train from drawmaha_solver.deepcfr.train import ( GRID, @@ -38,6 +38,7 @@ DeepRun, Fitted, OracleLearner, + OracleModel, current_iterate, load_run, mark_name, @@ -82,8 +83,8 @@ def __init__(self, inner: Collector) -> None: self.events: list[tuple] = [] # -- the learner -- - def fit(self, rows, t, targets, *, seed) -> Fitted: - self.events.append(("fit", seed, (rows.copy(), t.copy(), targets.copy()))) + def fit(self, rows, t, targets, weight, *, seed) -> Fitted: + self.events.append(("fit", seed, (rows.copy(), t.copy(), targets.copy(), weight.copy()))) return Fitted(("model", seed), None) def write(self, table, models) -> None: @@ -248,8 +249,8 @@ def test_fit_receives_the_memory_as_it_stands_after_the_merge(collector): step(run, spy, spy) fits = spy.of("fit") for seat in SEATS: - _, (rows, t, targets) = [fit for fit in fits if fit[0][2] == seat][-1] - for given, resident in zip((rows, t, targets), run.advantage[seat].contents(), strict=True): + _, (rows, t, targets, weight) = [fit for fit in fits if fit[0][2] == seat][-1] + for given, resident in zip((rows, t, targets, weight), run.advantage[seat].contents(), strict=True): assert given.dtype == resident.dtype np.testing.assert_array_equal(given, resident) assert rows.shape[0] == run.advantage[seat].size > 0 @@ -373,11 +374,40 @@ def test_oracle_models_round_trip(tmp_path, collector): step(run, collector, learner) learner.save(run.nets[0], tmp_path / "p0-t00001.pt") back = learner.load(tmp_path / "p0-t00001.pt") - for name in ("rows", "t", "targets"): + for name in ("rows", "t", "targets", "weight"): a, b = getattr(run.nets[0], name), getattr(back, name) assert a.dtype == b.dtype np.testing.assert_array_equal(a, b) assert run.nets[0].rows.shape[0] == run.advantage[0].size + assert run.nets[0].weight.dtype == np.float32 and (run.nets[0].weight == 1.0).all() + +def test_an_oracle_model_saved_without_a_weight_loads_with_ones(tmp_path): + old = tmp_path / "p0-t00003.pt" + with old.open("wb") as file: + np.savez(file, rows=np.arange(4, dtype=np.int32), t=np.full(4, 3, np.int32), targets=np.ones((4, TARGET_WIDTH), np.float32)) + back = OracleLearner().load(old) + assert back.weight.dtype == np.float32 and back.weight.tolist() == [1.0] * 4 + +def oracle_table(*models: OracleModel) -> np.ndarray: + table = PackedTable.whole_game() + OracleLearner().write(table, models) + return table.cumulative_regret + +def test_the_oracle_weighs_a_row_by_its_weight(): + # One real row with the sample r = (0.5, −0.25, 0, 0) at t = 3. A weight + # of 2 is the row offered twice; a weight of 0 is the row never offered. + row = np.array([first_row_at(((),))] * 2, dtype=np.int32) + t = np.array([3, 3], dtype=np.int32) + r = np.array([[0.5, -0.25, 0.0, 0.0]] * 2, dtype=np.float32) + w = lambda *values: np.array(values, dtype=np.float32) # noqa: E731 + twice = oracle_table(OracleModel(row, t, r, w(1.0, 1.0))) + weighted = oracle_table(OracleModel(row[:1], t[:1], r[:1], w(2.0))) + np.testing.assert_array_equal(twice, weighted) + assert twice.any() + once = oracle_table(OracleModel(row[:1], t[:1], r[:1], w(1.0))) + with_an_inert_row = oracle_table(OracleModel(row, t, r, w(1.0, 0.0))) + np.testing.assert_array_equal(once, with_an_inert_row) + assert not np.array_equal(once, twice) # --------------------------------------------------------------------------- # The resume state diff --git a/tests/deepcfr/test_train_torch.py b/tests/deepcfr/test_train_torch.py index 3e0ce5e5..44f9d785 100644 --- a/tests/deepcfr/test_train_torch.py +++ b/tests/deepcfr/test_train_torch.py @@ -107,15 +107,15 @@ def test_fit_reports_a_finite_loss_and_a_run_logs_it(tmp_path): run = new_run(TINY) with Collector(seed=9, workers=0) as collector: rows = collector.collect(seat=0, t=1, walks=5) - fitted = learner.fit(rows.advantage_rows, np.ones(len(rows.advantage_rows), np.int32), rows.advantage_targets, seed=(9, 1, 0)) + ones = np.ones(len(rows.advantage_rows), np.int32) + fitted = learner.fit(rows.advantage_rows, ones, rows.advantage_targets, ones.astype(np.float32), seed=(9, 1, 0)) assert isinstance(fitted.loss, float) and math.isfinite(fitted.loss) - again = learner.fit(rows.advantage_rows, np.ones(len(rows.advantage_rows), np.int32), rows.advantage_targets, seed=(9, 1, 0)) + again = learner.fit(rows.advantage_rows, ones, rows.advantage_targets, ones.astype(np.float32), seed=(9, 1, 0)) assert again.loss == fitted.loss # the held batch is a function of the fit's seed # ... but not the fit's own first minibatch, which the net has just trained on. from drawmaha_solver.deepcfr.fit import _fold_seed, batch_loss first = torch.randint(len(rows.advantage_rows), (TINY.batch,), generator=torch.Generator().manual_seed(_fold_seed((9, 1, 0)))).numpy() - ones = np.ones(len(rows.advantage_rows), np.int32) with torch.inference_mode(): trained_on = float(batch_loss(fitted.model, rows.advantage_rows[first], ones[first], rows.advantage_targets[first]).item()) assert fitted.loss != trained_on @@ -140,23 +140,29 @@ def test_the_targets_reaching_the_fit_are_the_pot_scaled_rule_targets(monkeypatc handed = {} real = fit.fit_from_scratch - def spy(rows, t, targets, *, seed, config): - handed.update(rows=rows, t=t, targets=targets) - return real(rows, t, targets, seed=seed, config=config) + def spy(rows, t, targets, weight, *, seed, config): + handed.update(rows=rows, t=t, targets=targets, weight=weight) + return real(rows, t, targets, weight, seed=seed, config=config) monkeypatch.setattr(fit, "fit_from_scratch", spy) - NetLearner(replace(TINY, pot_power=0.5)).fit(rows_, t, targets, seed=(9, 3, 0)) + # Weights distinct per row, so a dropped or reordered column would show. + weight = np.linspace(0.5, 2.0, len(t)).astype(np.float32) + NetLearner(replace(TINY, pot_power=0.5)).fit(rows_, t, targets, weight, seed=(9, 3, 0)) np.testing.assert_array_equal(handed["targets"], pot_scaled(rows_, targets, power=0.5)) assert not np.array_equal(handed["targets"], targets) np.testing.assert_array_equal(handed["t"], t) + # The weight is a sample weight and never a divisor of the target: the + # pot scaling leaves it exactly as the memory held it. + assert handed["weight"] is weight - NetLearner(replace(TINY, pot_power=0.5, rule="dcfr")).fit(rows_, t, targets, seed=(9, 3, 0)) + NetLearner(replace(TINY, pot_power=0.5, rule="dcfr")).fit(rows_, t, targets, weight, seed=(9, 3, 0)) ruled = rule_targets(targets, t, now=3, rule="dcfr") np.testing.assert_array_equal(handed["targets"], pot_scaled(rows_, ruled, power=0.5)) np.testing.assert_array_equal(handed["t"], np.ones(len(t), np.float32)) + assert handed["weight"] is weight - NetLearner(replace(TINY, pot_power=0.0)).fit(rows_, t, targets, seed=(9, 3, 0)) + NetLearner(replace(TINY, pot_power=0.0)).fit(rows_, t, targets, weight, seed=(9, 3, 0)) assert handed["targets"] is targets # power 0 hands the memory's own array through def test_pot_scaled_regrets_write_the_same_strategy_through_the_slot_rule_and_the_ledger(): From 394f9f7881e11286a65a697e59c2f7d7bd52282b Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 17:51:52 -0500 Subject: [PATCH 10/16] =?UTF-8?q?fix(rung4):=20a=20scalar=20rows=20or=20we?= =?UTF-8?q?ight=20and=20a=20corrupt=20pre-column=20file=20are=20refused=20?= =?UTF-8?q?by=20name=20=E2=80=94=20the=20ones=20default=20ran=20ahead=20of?= =?UTF-8?q?=20the=20checks=20it=20relied=20on;=20and=20the=20weight-one=20?= =?UTF-8?q?pin=20is=20against=20the=20old=20loss=20formula,=20not=20the=20?= =?UTF-8?q?code=20itself?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the fresh-context review before the push. Reservoir.load sized the default weight from a negative `offered` before refusing it; merge and fit_from_scratch sized it from a 0-d rows before the 1-D check; a 0-d weight on load indexed its own empty shape. Each now fails on the by-name ValueError the old code gave. test_weight_one_is_the_unweighted_fit_bit_for_bit compared the fit to itself (both paths built the same ones); it now restates the pre-column loss, t / mean(t) over the legal squared error, and pins batch_loss at weight one and by default against it to the bit, then the whole fit. --- src/drawmaha_solver/deepcfr/fit.py | 10 ++++---- src/drawmaha_solver/deepcfr/reservoir.py | 20 ++++++--------- src/drawmaha_solver/deepcfr/train.py | 2 +- tests/deepcfr/test_fit.py | 29 ++++++++++++++++++--- tests/deepcfr/test_reservoir.py | 32 ++++++++++++++++++++++++ 5 files changed, 72 insertions(+), 21 deletions(-) diff --git a/src/drawmaha_solver/deepcfr/fit.py b/src/drawmaha_solver/deepcfr/fit.py index 9bb4546e..a526b0c3 100644 --- a/src/drawmaha_solver/deepcfr/fit.py +++ b/src/drawmaha_solver/deepcfr/fit.py @@ -217,14 +217,14 @@ def fit_from_scratch( rows = np.asarray(rows) t = np.asarray(t) targets = np.asarray(targets, dtype=np.float32) - weight = np.ones(len(rows), dtype=np.float32) if weight is None else np.asarray(weight, dtype=np.float32) # A (n, 1) t would broadcast against the (n,) error into an (n, n) loss # and train on the wrong number without a word, so the shapes are # refused by name, not left to broadcasting. - if rows.ndim != 1 or t.ndim != 1 or weight.ndim != 1: - raise ValueError( - f"rows, t and weight must be 1-D, got shapes {rows.shape}, {t.shape} and {weight.shape}" - ) + if rows.ndim != 1 or t.ndim != 1: + raise ValueError(f"rows and t must be 1-D, got shapes {rows.shape} and {t.shape}") + weight = np.ones(len(rows), dtype=np.float32) if weight is None else np.asarray(weight, dtype=np.float32) + if weight.ndim != 1: + raise ValueError(f"weight must be 1-D, got shape {weight.shape}") if not np.issubdtype(rows.dtype, np.integer): raise ValueError(f"rows must be integer row numbers, got dtype {rows.dtype}") if not len(rows) == len(t) == len(targets) == len(weight): diff --git a/src/drawmaha_solver/deepcfr/reservoir.py b/src/drawmaha_solver/deepcfr/reservoir.py index f7315baf..950393d5 100644 --- a/src/drawmaha_solver/deepcfr/reservoir.py +++ b/src/drawmaha_solver/deepcfr/reservoir.py @@ -228,19 +228,15 @@ def load(cls, path: Path) -> Reservoir: raise ValueError(f"{path}: capacity must be a positive int, got {capacity}") memory = cls(capacity) size = min(offered, capacity) - if weight is None: - weight = np.ones(size, dtype=np.float32) - if ( - offered < 0 - or rows.shape != (size,) - or t.shape != (size,) - or targets.shape != (size, TARGET_WIDTH) - or weight.shape != (size,) - ): + if offered < 0 or rows.shape != (size,) or t.shape != (size,) or targets.shape != (size, TARGET_WIDTH): raise ValueError( - f"{path} holds {rows.shape[0]} rows and {weight.shape[0]} weights, but " - f"{offered} offers into a capacity of {capacity} means {size} resident" + f"{path} holds {rows.shape} rows, but {offered} offers into a capacity " + f"of {capacity} means {size} resident" ) + if weight is None: + weight = np.ones(size, dtype=np.float32) + elif weight.shape != (size,): + raise ValueError(f"{path} holds {size} rows but a weight of shape {weight.shape}") # The arrays are poured in as they are: a float `rows` would be # truncated into the int32 slots without a word. expected = ( @@ -273,11 +269,11 @@ def _validate_offer( """The phase's rows as int32, its targets and weights as float32, or a ValueError naming the fault.""" rows = np.asarray(rows) targets = np.asarray(targets) - weight = np.ones(rows.shape[0], dtype=np.float32) if weight is None else np.asarray(weight) # A row is a table row number: a float or a negative is a caller handing # the wrong array. if rows.ndim != 1 or not np.issubdtype(rows.dtype, np.integer): raise ValueError(f"rows must be a 1-D integer array, got {rows.dtype} {rows.shape}") + weight = np.ones(rows.shape[0], dtype=np.float32) if weight is None else np.asarray(weight) if rows.size and rows.min() < 0: raise ValueError(f"rows must not be negative, got {int(rows.min())}") # Rows are stored as int32; a bigger one would wrap silently in `astype`. diff --git a/src/drawmaha_solver/deepcfr/train.py b/src/drawmaha_solver/deepcfr/train.py index 891ada27..7dbbccf8 100644 --- a/src/drawmaha_solver/deepcfr/train.py +++ b/src/drawmaha_solver/deepcfr/train.py @@ -37,7 +37,7 @@ with it Deep CFR collapses into rung 3's tabular LCFR. The weight is the memory's fourth column (`reservoir.py`): 1 for every row a walk offers, so until the targeted-sampling plan's boost stage injects rows that stand for -more or fewer samples than one, it multiplies nothing. The one thing the loop asks +more or fewer samples than one, it changes nothing. The one thing the loop asks of a `write` is that it assigns INTO `table.cumulative_regret` and never rebinds it: the collector's workers read that array through a shared block, and a rebound array would leave them walking stale regrets. diff --git a/tests/deepcfr/test_fit.py b/tests/deepcfr/test_fit.py index 031feb64..57fa0a89 100644 --- a/tests/deepcfr/test_fit.py +++ b/tests/deepcfr/test_fit.py @@ -101,10 +101,30 @@ def test_later_iterations_weigh_more(): ) np.testing.assert_allclose(legal_prediction(net, row), 0.5, atol=0.05) -def test_weight_one_is_the_unweighted_fit_bit_for_bit(): - # t · 1.0 is t exactly in float32, so the loss, the gradients and the - # weights are the same bits as a fit told nothing about weights. +def test_weight_one_is_the_loss_before_the_column_existed_bit_for_bit(): + # The loss as it was written before the weight column — the squared + # error over the legal columns, each row scaled by t / mean(t) — restated + # here so the pin is against the formula, not against the code under + # test. t · 1.0 is t exactly in float32, so the two agree to the bit. rows, t, targets = real_rows(300, seed=4) + torch.manual_seed(0) + net = Net() + built = layout() + point, _ = built.locate_rows(rows) + gather = torch.from_numpy(legal_gather()[point].astype(np.int64)) + legal = gather != NO_ACTION + with torch.no_grad(): + outputs = net(torch.from_numpy(built.features(rows))) + predicted = outputs.gather(1, gather.clamp(min=0)) + target = torch.where(legal, torch.from_numpy(targets), 0.0) + error = torch.where(legal, (predicted - target) ** 2, 0.0).sum(dim=1) + t32 = torch.from_numpy(t.astype(np.float32)) + before = ((t32 / t32.mean()) * error).mean() + now = batch_loss(net, rows, t, targets, np.ones(300, np.float32)) + default = batch_loss(net, rows, t, targets) + assert torch.equal(now, before) and torch.equal(default, before) + # And through the whole fit: the weights a weight-one fit lands on are + # the ones a fit told nothing about weights lands on. plain = fit_from_scratch(rows, t, targets, seed=(0, 1, 0), config=SMALL) ones = fit_from_scratch(rows, t, targets, np.ones(300, np.float32), seed=(0, 1, 0), config=SMALL) for a, b in zip(plain.parameters(), ones.parameters(), strict=True): @@ -304,6 +324,9 @@ def test_fit_refuses_rows_or_t_that_are_not_one_dimensional_ints(): fit_from_scratch(rows[:, None], t, targets, seed=(0,), config=SMALL) with pytest.raises(ValueError, match="integer"): fit_from_scratch(rows.astype(np.float64), t, targets, seed=(0,), config=SMALL) + # A scalar: the ones default for the weight must not trip over it first. + with pytest.raises(ValueError, match="1-D"): + fit_from_scratch(np.int64(rows[0]), t[:1], targets[:1], seed=(0,), config=SMALL) # --------------------------------------------------------------------------- # The GPU path diff --git a/tests/deepcfr/test_reservoir.py b/tests/deepcfr/test_reservoir.py index 5d71c17a..4df815d7 100644 --- a/tests/deepcfr/test_reservoir.py +++ b/tests/deepcfr/test_reservoir.py @@ -216,6 +216,9 @@ def test_the_stream_is_a_function_of_its_four_words(): def test_merge_refuses_bad_rows(): rng0 = np.random.default_rng(0) memory = Reservoir(capacity=4) + # A scalar: the ones default for the weight must not trip over it first. + with pytest.raises(ValueError, match="1-D"): + memory.merge(np.int64(3), targets(1), t=1, rng=rng0) with pytest.raises(ValueError, match="same length"): memory.merge(np.arange(3), targets(2), t=1, rng=rng0) with pytest.raises(ValueError, match="4"): @@ -344,6 +347,35 @@ def write(name, weight): with pytest.raises(ValueError, match="weight must be float32"): Reservoir.load(write("double.npz", np.ones(3, np.float64))) +def test_load_refuses_a_corrupt_file_without_a_weight_by_name(tmp_path): + # The ones default must not run ahead of the counts check: a negative + # `offered` in a pre-column file is refused naming the file, as before. + broken = tmp_path / "negative.npz" + np.savez( + broken, + rows=np.zeros(0, np.int32), + t=np.zeros(0, np.int32), + targets=np.zeros((0, TARGET_WIDTH), np.float32), + offered=np.int64(-1), + capacity=np.int64(8), + ) + with pytest.raises(ValueError, match="negative.npz"): + Reservoir.load(broken) + +def test_load_refuses_a_scalar_weight_by_name(tmp_path): + scalar = tmp_path / "scalar.npz" + np.savez( + scalar, + rows=np.arange(3, dtype=np.int32), + t=np.ones(3, np.int32), + targets=targets(3), + weight=np.float32(1.0), + offered=np.int64(3), + capacity=np.int64(8), + ) + with pytest.raises(ValueError, match="scalar.npz"): + Reservoir.load(scalar) + def test_a_loaded_memory_can_keep_filling_to_its_capacity(tmp_path): memory = filled_memory() memory.save(tmp_path / "m.npz") From 11b6e7bd115bb6ff172ec57fb8f92ace223fbe94 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Thu, 5 Nov 2026 18:14:08 -0500 Subject: [PATCH 11/16] =?UTF-8?q?fix(rung4):=20a=20weight=20is=20checked?= =?UTF-8?q?=20after=20its=20cast=20to=20float32,=20and=20t=20=C3=97=20weig?= =?UTF-8?q?ht=20for=20overflow,=20on=20merge,=20on=20load=20and=20in=20the?= =?UTF-8?q?=20fit=20=E2=80=94=20a=20float64=20weight=20past=20float32's=20?= =?UTF-8?q?range=20passed=20the=20finite=20check=20and=20was=20stored=20as?= =?UTF-8?q?=20inf;=20and=20the=20oracle's=20"offered=20twice"=20claim=20ho?= =?UTF-8?q?lds=20to=20the=20bit=20only=20when=20the=20two=20are=20alone=20?= =?UTF-8?q?at=20the=20column?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `_validate_offer` casts to float32 first and checks finite, non-negative and t × weight finite after; a bool or non-numeric weight is refused by name. - `Reservoir.load` refuses a weight that is NaN, negative or overflows t × weight, naming the file, as `merge` never writes one. - `fit_from_scratch` refuses a t × weight that overflows float32, the loss's scale. - `OracleLearner` docstring: weight 2 equals the row offered twice exactly only when nothing else sits at the column in that t′; otherwise up to rounding, since `np.add.at` sums one at a time. A new test pins two rows of widths 2 and 3 with weights 3 and 2 against the rows offered three and two times, so a weight landing on a neighbour's columns would show. - `batch_loss` says why it defaults the weight while `Learner.fit` does not; `NetLearner.fit` notes the held loss is a log line nothing decides on. --- src/drawmaha_solver/deepcfr/fit.py | 25 +++++++++---- src/drawmaha_solver/deepcfr/reservoir.py | 45 +++++++++++++++++------- src/drawmaha_solver/deepcfr/train.py | 13 +++++-- tests/deepcfr/test_fit.py | 6 ++++ tests/deepcfr/test_reservoir.py | 39 +++++++++++++++++++- tests/deepcfr/test_train.py | 24 +++++++++++++ 6 files changed, 129 insertions(+), 23 deletions(-) diff --git a/src/drawmaha_solver/deepcfr/fit.py b/src/drawmaha_solver/deepcfr/fit.py index a526b0c3..cba13b68 100644 --- a/src/drawmaha_solver/deepcfr/fit.py +++ b/src/drawmaha_solver/deepcfr/fit.py @@ -136,9 +136,12 @@ def batch_loss( Public so a test and PR 4's logging read the same number the optimiser does. Builds the batch's features on the fly from `layout()`. `weight` - left out is a batch of plain samples, every row weighing 1. Assumes - what `fit_from_scratch` checks: every t is >= 1 and every weight is - finite and non-negative. + left out is a batch of plain samples, every row weighing 1: a default + here, where a caller hands arrays it built itself, while `Learner.fit` + in `train.py` has none, because that is where a memory's column could + be dropped by accident. Assumes what `fit_from_scratch` checks: every t + is >= 1, every weight finite and non-negative, and t × weight within + float32's range. """ built = layout() point, _ = built.locate_rows(rows) @@ -207,7 +210,8 @@ def fit_from_scratch( - a `rows`, `t` or `weight` that is not 1-D, or non-integer rows; - mismatched lengths, or an empty memory; - a non-finite t, or one below 1; - - a non-finite or negative weight, or weights with no positive entry + - a non-finite or negative weight, one whose t × weight overflows + float32 (the loss's scale), or weights with no positive entry (nothing to fit, as an empty memory is); - targets not `MAX_WIDTH` wide, or non-finite at a legal column; - a row outside the table. @@ -222,7 +226,8 @@ def fit_from_scratch( # refused by name, not left to broadcasting. if rows.ndim != 1 or t.ndim != 1: raise ValueError(f"rows and t must be 1-D, got shapes {rows.shape} and {t.shape}") - weight = np.ones(len(rows), dtype=np.float32) if weight is None else np.asarray(weight, dtype=np.float32) + with np.errstate(over="ignore"): # a weight past float32's range is refused below as the inf it became + weight = np.ones(len(rows), dtype=np.float32) if weight is None else np.asarray(weight, dtype=np.float32) if weight.ndim != 1: raise ValueError(f"weight must be 1-D, got shape {weight.shape}") if not np.issubdtype(rows.dtype, np.integer): @@ -240,8 +245,16 @@ def fit_from_scratch( raise ValueError("every t must be finite") if t.min() < 1: raise ValueError(f"every t must be an iteration number >= 1, got a minimum of {t.min()}") + # Checked after the cast to float32, so a float64 weight past float32's + # range is caught as the inf it became, not passed as the finite number + # it was given as. if not np.isfinite(weight).all() or weight.min() < 0: - raise ValueError("every weight must be finite and non-negative") + raise ValueError("every weight must be finite and non-negative in float32") + # The loss scales a row by t × weight in float32; a product past its + # range would reach the batch as inf and the loss as NaN. + with np.errstate(over="ignore"): + if not np.isfinite(t.astype(np.float32) * weight).all(): + raise ValueError("t × weight must fit float32") # Every row inert: the batches would all be the weightless case the # loss guards, and 4,000 steps of zero gradient would "fit" nothing. if not (weight > 0).any(): diff --git a/src/drawmaha_solver/deepcfr/reservoir.py b/src/drawmaha_solver/deepcfr/reservoir.py index 950393d5..a51149c7 100644 --- a/src/drawmaha_solver/deepcfr/reservoir.py +++ b/src/drawmaha_solver/deepcfr/reservoir.py @@ -139,8 +139,9 @@ def merge( ) -> None: """Offer one phase's rows, all stamped `t`, drawing from `rng`. - `weight` is one finite, non-negative float per row; left out, every - row weighs 1, the walk's own sample. `rng` needs + `weight` is one finite, non-negative float per row, with t × weight + within float32's range; left out, every row weighs 1, the walk's own + sample. `rng` needs `Generator.integers(low, high)` with an array `high`: one draw per late offer. Rows below capacity are kept outright and draw nothing, so a memory still filling consumes no randomness. @@ -213,7 +214,12 @@ def load(cls, path: Path) -> Reservoir: Refuses a file that is not a reservoir (no `offered`) and one whose resident rows do not match its counts, naming the file either way. A file with no `weight` is from before the column existed: its rows - were all plain samples and load weighing one. + were all plain samples and load weighing one. A `weight` the column + would not have stored — not float32, not one per row, negative, + non-finite, or overflowing t × weight in float32 — is refused naming + the file: `merge` never writes one, so it is a corrupt or hand-edited + file, and `fit_from_scratch` would otherwise be the first to say so, + without the file's name. """ path = Path(path) with np.load(path, allow_pickle=False) as saved: @@ -248,6 +254,12 @@ def load(cls, path: Path) -> Reservoir: for name, array, dtype in expected: if array.dtype != dtype: raise ValueError(f"{path}: {name} must be {dtype.__name__}, got {array.dtype}") + # What `merge` refuses on the way in, refused on the way back as well. + if not (np.isfinite(weight).all() and (weight >= 0).all()): + raise ValueError(f"{path}: every weight must be finite and non-negative") + with np.errstate(over="ignore"): + if not np.isfinite(t.astype(np.float32) * weight).all(): + raise ValueError(f"{path}: t × weight must fit float32") memory._rows[:size] = rows memory._t[:size] = t memory._targets[:size] = targets @@ -290,11 +302,10 @@ def _validate_offer( ) if weight.shape != (rows.shape[0],): raise ValueError(f"weight must be shaped ({rows.shape[0]},), one per row, got {weight.shape}") - # The refit multiplies a row's error by its weight: a negative one would - # pull the net AWAY from the row's target, and a NaN would spread through - # the batch's mean into every row's gradient. - if weight.size and not (np.isfinite(weight).all() and weight.min() >= 0): - raise ValueError("every weight must be finite and non-negative") + # A weight is a count of samples: a bool, a complex or an object array is + # a caller handing the wrong array, not a count. + if not (np.issubdtype(weight.dtype, np.floating) or np.issubdtype(weight.dtype, np.integer)): + raise ValueError(f"weight must be a float or int array, got dtype {weight.dtype}") # Iterations start at 1; the refit weights a row by its t, so 0 would # erase it and a negative would pull against the others. if not _is_int(t) or t < 1: @@ -303,8 +314,16 @@ def _validate_offer( # below the bound, which is here so the stamp can never wrap in silence. if t >= 2**31: raise ValueError(f"t must fit int32, got {t!r}") - return ( - rows.astype(np.int32, copy=False), - targets.astype(np.float32, copy=False), - weight.astype(np.float32, copy=False), - ) + # The column is float32, so the weight is checked AFTER the cast: a + # float64 weight past float32's range is finite as given and inf once + # stored. The refit multiplies a row's error by t × weight in float32: a + # negative weight would pull the net AWAY from the row's target, a NaN + # or an inf (the product's, if not the weight's own) would spread through + # the batch's mean into every row's gradient. + with np.errstate(over="ignore"): # the overflow is what is being looked for + weight = weight.astype(np.float32, copy=False) + if weight.size and not (np.isfinite(weight).all() and weight.min() >= 0): + raise ValueError("every weight must be finite and non-negative in float32") + if weight.size and not np.isfinite(np.float32(t) * weight).all(): + raise ValueError(f"t × weight must fit float32: at t = {t} a weight of {weight.max()} overflows") + return rows.astype(np.int32, copy=False), targets.astype(np.float32, copy=False), weight diff --git a/src/drawmaha_solver/deepcfr/train.py b/src/drawmaha_solver/deepcfr/train.py index 7dbbccf8..d6f0ddac 100644 --- a/src/drawmaha_solver/deepcfr/train.py +++ b/src/drawmaha_solver/deepcfr/train.py @@ -1078,6 +1078,9 @@ def fit( net = fit_from_scratch(rows, t, targets, weight, seed=seed, config=config) draw = torch.Generator().manual_seed(_fold_seed((*seed, _HELD_TAG))) held = torch.randint(len(rows), (config.batch,), generator=draw).numpy() + # The held loss is a number for the progress line (`StepStats.loss`) + # and nothing decides on it: a held batch of weight-0 rows alone + # reads 0, as the loss would, and says so there rather than NaN. with torch.inference_mode(): loss = float(batch_loss(net, rows[held], t[held], targets[held], weight[held]).item()) return Fitted(net, loss) @@ -1156,9 +1159,13 @@ class OracleLearner: memory order with `np.add.at`, then `R[cols] += t′ · totals` — so the bits match `bank_rows` through any number of iterations, not only the first: × 1.0 is exact, so a memory of weight-one rows banks what it - banked before the column existed. A row of weight 2 banks what the same - row offered twice would (r + r = 2r exactly), and a row of weight 0 - banks nothing. The two models are summed one after the other, which is + banked before the column existed. A row of weight w banks w × r, so a + row of weight 0 banks nothing, and a row of weight 2 banks 2r, which is + what the same row offered twice banks when the two are the only samples + at the column in that t′ (0 + r + r is 2r exactly); with other samples + there, the two agree only up to rounding, since `np.add.at` sums the + column's samples one at a time and (a + r) + r is not a + 2r to the + bit. The two models are summed one after the other, which is only LCFR's order because a row belongs to one acting seat and so appears in one model. A row no sample names stays exactly 0.0. NumPy only, so gate 1 runs without torch. diff --git a/tests/deepcfr/test_fit.py b/tests/deepcfr/test_fit.py index 57fa0a89..e5ada02c 100644 --- a/tests/deepcfr/test_fit.py +++ b/tests/deepcfr/test_fit.py @@ -179,6 +179,12 @@ def test_fit_refuses_bad_weights(): fit_from_scratch(rows, t, targets, np.ones((10, 1)), seed=(0,), config=SMALL) with pytest.raises(ValueError, match="nothing to fit"): fit_from_scratch(rows, t, targets, np.zeros(10), seed=(0,), config=SMALL) + # Finite in float64, inf in the float32 the loss computes in. + with pytest.raises(ValueError, match="finite"): + fit_from_scratch(rows, t, targets, np.r_[np.ones(9), 1e300], seed=(0,), config=SMALL) + # Finite in float32 on its own; t × weight is not, and that is the scale. + with pytest.raises(ValueError, match="fit float32"): + fit_from_scratch(rows, np.full(10, 2), targets, np.r_[np.ones(9), 2e38], seed=(0,), config=SMALL) def test_loss_is_over_the_legal_columns_only(): # A 2-wide betting row: its loss is the sum over its two columns, so a diff --git a/tests/deepcfr/test_reservoir.py b/tests/deepcfr/test_reservoir.py index 4df815d7..932d3d68 100644 --- a/tests/deepcfr/test_reservoir.py +++ b/tests/deepcfr/test_reservoir.py @@ -151,7 +151,19 @@ def test_merge_refuses_bad_weights(): memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 1.0, 1.0])) with pytest.raises(ValueError, match="weight"): memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.ones((2, 1))) - assert memory.offered == 0 # a refused offer changes nothing + # Finite as given, inf once cast into the float32 column: refused as the + # inf it would be stored as, not passed as the float64 it was handed as. + with pytest.raises(ValueError, match="finite"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 1e300])) + # The refit's scale is t × weight in float32: a weight the column holds + # but the product cannot is refused at the t that overflows it. + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([1.0, 2e38], np.float32)) + with pytest.raises(ValueError, match="fit float32"): + memory.merge(np.arange(2), targets(2), t=2, rng=rng, weight=np.array([1.0, 2e38], np.float32)) + # A bool array is a caller handing the wrong array, not a count of samples. + with pytest.raises(ValueError, match="float or int"): + memory.merge(np.arange(2), targets(2), t=1, rng=rng, weight=np.array([True, True])) + assert memory.offered == 2 # the one accepted offer counts; a refused one changes nothing def test_an_empty_merge_changes_nothing(): memory = filled_memory() @@ -347,6 +359,31 @@ def write(name, weight): with pytest.raises(ValueError, match="weight must be float32"): Reservoir.load(write("double.npz", np.ones(3, np.float64))) +def test_load_refuses_a_weight_with_a_bad_value_by_name(tmp_path): + # `merge` never stores one, so a NaN, a negative, or a weight whose + # t × weight overflows float32 is a corrupt or hand-edited file, refused + # by name here rather than by `fit_from_scratch` later without it. + def write(name, weight, t=1): + path = tmp_path / name + np.savez( + path, + rows=np.arange(3, dtype=np.int32), + t=np.full(3, t, np.int32), + targets=targets(3), + weight=np.array(weight, np.float32), + offered=np.int64(3), + capacity=np.int64(8), + ) + return path + + with pytest.raises(ValueError, match="nan.npz.*finite"): + Reservoir.load(write("nan.npz", [1.0, np.nan, 1.0])) + with pytest.raises(ValueError, match="negative.npz.*non-negative"): + Reservoir.load(write("negative.npz", [1.0, -1.0, 1.0])) + with pytest.raises(ValueError, match="overflow.npz.*fit float32"): + Reservoir.load(write("overflow.npz", [1.0, 2e38, 1.0], t=2)) + assert Reservoir.load(write("fine.npz", [1.0, 2e38, 1.0], t=1)).contents()[3][1] == np.float32(2e38) + def test_load_refuses_a_corrupt_file_without_a_weight_by_name(tmp_path): # The ones default must not run ahead of the counts check: a negative # `offered` in a pre-column file is refused naming the file, as before. diff --git a/tests/deepcfr/test_train.py b/tests/deepcfr/test_train.py index 8047deb7..c1d57c36 100644 --- a/tests/deepcfr/test_train.py +++ b/tests/deepcfr/test_train.py @@ -396,6 +396,9 @@ def oracle_table(*models: OracleModel) -> np.ndarray: def test_the_oracle_weighs_a_row_by_its_weight(): # One real row with the sample r = (0.5, −0.25, 0, 0) at t = 3. A weight # of 2 is the row offered twice; a weight of 0 is the row never offered. + # The equality is to the bit because these are the only samples at the + # column: 0 + r + r is 2r exactly. Beside other samples the two would + # agree only up to rounding (see the `OracleLearner` docstring). row = np.array([first_row_at(((),))] * 2, dtype=np.int32) t = np.array([3, 3], dtype=np.int32) r = np.array([[0.5, -0.25, 0.0, 0.0]] * 2, dtype=np.float32) @@ -409,6 +412,27 @@ def test_the_oracle_weighs_a_row_by_its_weight(): np.testing.assert_array_equal(once, with_an_inert_row) assert not np.array_equal(once, twice) +def test_the_oracle_keeps_each_weight_on_its_own_row_across_widths(): + # Two rows of different widths in one model — the opening spot, 2 wide + # (check or pot, no bet to fold to), and the spot after pot, pot, 3 wide + # — each with its own sample and weight. `write` flattens the samples + # row by row through `legal` and repeats each weight by its row's width; + # a weight landing on a neighbour's columns would make the table differ + # from the rows offered weight-many times. Exact as above: 0 + r + r + r + # and 3r are each one rounding of the same real number. + rows = np.array([first_row_at(((),)), first_row_at(((Action.POT, Action.POT),))], dtype=np.int32) + widths = layout().width[layout().locate_rows(rows.astype(np.int64))[0]] + assert widths.tolist() == [2, 3] + t = np.array([3, 3], dtype=np.int32) + r = np.array([[0.5, -0.25, 0.0, 0.0], [-0.75, 0.375, 0.125, 0.0]], dtype=np.float32) + weighted = oracle_table(OracleModel(rows, t, r, np.array([3.0, 2.0], np.float32))) + repeated = oracle_table( + OracleModel(np.repeat(rows, [3, 2]), np.repeat(t, [3, 2]), np.repeat(r, [3, 2], axis=0), np.ones(5, np.float32)) + ) + np.testing.assert_array_equal(weighted, repeated) + swapped = oracle_table(OracleModel(rows, t, r, np.array([2.0, 3.0], np.float32))) + assert not np.array_equal(weighted, swapped) + # --------------------------------------------------------------------------- # The resume state # --------------------------------------------------------------------------- From 081f756913e3a2de7222acd8a6726a083df1ae70 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Fri, 6 Nov 2026 04:29:29 -0500 Subject: [PATCH 12/16] =?UTF-8?q?feat(rung4):=20the=20boost=20target=20sam?= =?UTF-8?q?ples=20k=20of=20n=20of=20every=20source=20below=20a=20high-nois?= =?UTF-8?q?e=20row=20=E2=80=94=20board=20card,=20both=20seats'=20replaceme?= =?UTF-8?q?nts,=20the=20opponent's=20actions=20and=20range=20=E2=80=94=20e?= =?UTF-8?q?ach=20unbiased=20against=20the=20exact=20sum,=20which=20stays?= =?UTF-8?q?=20as=20EXACT,=20with=20the=20arrival=20always=20exact=20and=20?= =?UTF-8?q?one=20stream=20per=20(seed,=20t,=20seat,=20row)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit exact_target.py becomes boost_target.py (ExactTargets → BoostTargets). targets() takes a required Sampling; EXACT reproduces the previous output bit for bit. --- ...026-11-06-boost-target-modular-sampling.md | 95 +++++ src/drawmaha_solver/minidrawmaha/AGENTS.md | 11 +- .../{exact_target.py => boost_target.py} | 328 ++++++++++++++---- ...t_exact_target.py => test_boost_target.py} | 231 ++++++++++-- 4 files changed, 568 insertions(+), 97 deletions(-) create mode 100644 docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md rename src/drawmaha_solver/minidrawmaha/solver/{exact_target.py => boost_target.py} (72%) rename tests/minidrawmaha/{test_exact_target.py => test_boost_target.py} (74%) diff --git a/docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md b/docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md new file mode 100644 index 00000000..40504a61 --- /dev/null +++ b/docs/superpowers/plans/2026-11-06-boost-target-modular-sampling.md @@ -0,0 +1,95 @@ +# Boost target: modular sampling from a high-noise row — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Turn PR #89's exact target into a sampling framework. At a targeted row, it runs M extra play-outs from that row. Each source of randomness below the row is sampled k-of-n or enumerated, chosen per call. The result is unbiased for the regret the walk's samples estimate. Exact enumeration remains as the all-enumerate special case and the test oracle. + +**Architecture:** One value type, `Sampling`, names a count per source: opponent range, board cards, opponent draws, hero draws, opponent actions, and the number of play-outs. The existing downward recursion (`_value`, `_action_values`, `_push`) takes a private `_Draws` object. That object either returns everything (exact) or a random subset with its unbiasing scale. The upward back-solve (`_reach`) and the arrival stay exact always: the arrival is the cheap half, and an exact denominator under a sampled numerator keeps the ratio unbiased. Randomness is one `default_rng([seed, t, seat, row, TAG])` per row, so a row's result never depends on how rows are batched or split across workers. + +**Tech Stack:** Python 3.12, NumPy 2.5 (`Generator.multinomial` with 3-D `pvals`), pytest, `uv`. + +**Spec:** the chat design agreed 2026-11-06 (Alec): "sample extra starting from node x because node x is high variance … modular … board, opponent draws, hero draws, opp actions … unbiased". This builds on `~/Desktop/Claude/poker/targeted-sampling-plan/targeted-sampling-plan.md` §3 (row weight K · arrival · t, unchanged). + +## Global Constraints + +- The module imports `rules/` and the packed table only: never `exploitability.py`, `deepcfr`, or torch. The static import test stays. +- `Sampling` fields: `playouts: int = 1`, `opponent_range: int | None = None`, `board_cards: int | None = 1`, `opponent_draws: int | None = 1`, `hero_draws: int | None = 1`, `opponent_actions: int | None = 1`. `None` means enumerate all. `EXACT = Sampling(1, None, None, None, None, None)`. +- Chance sources (board, opponent draws, hero draws) sample k of n **without** replacement with scale n/k; k ≥ n is enumeration. +- Opponent actions are k i.i.d. draws from σ per opponent state, weighted count/k; `None` means split by σ. +- Opponent range is k i.i.d. hands per entry, drawn from that entry's compatible weight, each carrying W/k; `None` means the whole range. +- The arrival is always exact; `targets()` returns the same arrival as `arrivals()`, bit for bit. +- `targets(..., sampling=EXACT)` equals the pre-change output bit for bit. +- Stream: `np.random.default_rng([seed, t, seat, row, _STREAM_TAG])`; a non-EXACT `sampling` without `seed` and `t` is refused by name. +- Rename: `solver/exact_target.py` → `solver/boost_target.py`, `ExactTargets` → `BoostTargets`, `tests/minidrawmaha/test_exact_target.py` → `test_boost_target.py`; update `minidrawmaha/AGENTS.md`. +- The default suite stays torch-free and under its current time budget. Sampled unbiasedness tests use the restricted `SMALL`/`WIDE` deal sets. + +## Review Focus + +1. A sampled dimension that never actually samples, because the row has nothing of that kind below it. Every unbiasedness case must also assert that the sampled spread is > 0. +2. Hero-draw sampling must not leak into the hero's own **actions**. Hero actions are always enumerated under the hero's mix; only the replacement card is sampled. +3. Pruning a zero-weight opponent branch must not change the EXACT path. This is pinned by the bit-for-bit test. +4. `opponent_range = k` at a row whose compatible weight is zero (an unreachable row) must return zero regret, not NaN or a sampling error. +5. The same row requested twice in one call, or in two calls with rows in a different order, must give identical sampled results. + +--- + +### Task 1: rename and the `Sampling` value type + +**Files:** +- Rename: `src/drawmaha_solver/minidrawmaha/solver/exact_target.py` → `boost_target.py`; `tests/minidrawmaha/test_exact_target.py` → `test_boost_target.py` +- Modify: `src/drawmaha_solver/minidrawmaha/AGENTS.md` (the `exact_target` line) + +**Interfaces — Produces:** `Sampling` (frozen dataclass, fields above), `EXACT`, `BoostTargets` (the renamed class). `BoostTargets.targets(sigma, *, seat, rows, sampling: Sampling, seed: int | None = None, t: int | None = None) -> tuple[np.ndarray, np.ndarray]`. `sampling` is required, so every caller states its choice. + +- [ ] Step 1: Snapshot today's outputs (not committed). Save `targets` on `random_table(9)` for 40 rows spread over round-1, draw and round-2 points, both seats, to `.context/exact_snapshot.npz`. +- [ ] Step 2: `git mv` both files, update imports and the AGENTS.md line, and add `sampling=EXACT` at every existing test call site. Run `uv run pytest tests/minidrawmaha/test_boost_target.py tests/test_agent_docs.py`: PASS. +- [ ] Step 3: Failing test `test_sampling_refuses_counts_that_are_not_positive_ints`: `Sampling(board_cards=0)`, `Sampling(hero_draws=-1)`, `Sampling(opponent_actions=True)`, `Sampling(playouts=0)` and `Sampling(playouts=2, …all None)` each raise `ValueError`, and the message names the field. +- [ ] Step 4: Failing test `test_a_sampled_call_without_seed_and_t_is_refused`. +- [ ] Step 5: Implement `Sampling.__post_init__` and the refusal in `targets`, then run the tests: PASS. Commit `feat(rung4): the boost target names what it samples — one Sampling value, EXACT the oracle`. + +### Task 2: the `_Draws` seam and the exact path through it + +**Files:** `boost_target.py` (`_value`, `_action_values`, `_push`, `targets`) + +**Interfaces — Produces:** `_Draws(sampling, rng)` with: +- `board(kids: (nb,14)) -> (kids', scale)` +- `replacements(n_rows, fan=12) -> (pick | None, scale)`, used by both hero and opponent draws with their own field +- `actions(pvals: (nb,S,w)) -> weights (nb,S,w)` +- `range(u, mask, k) -> u'` + +On `EXACT`, every method is the identity, with no RNG and no array copy. + +- [ ] Step 1: Failing test `test_exact_sampling_reproduces_the_snapshot_bit_for_bit`. This is a temporary local test reading `.context/exact_snapshot.npz`; it's deleted before the commit, and its committed successor is Step 3 of this task. +- [ ] Step 2: Thread `draws: _Draws` through `_action_values` and `_value`, and give `_push` an optional `pick`. Skip an opponent child whose pushed weight has no positive entry (it returns zeros). Run: PASS bit for bit. +- [ ] Step 3: Committed test `test_a_count_at_or_above_the_fan_is_enumeration`: `Sampling(board_cards=14, opponent_draws=12, hero_draws=12, opponent_actions=None, opponent_range=None)` equals `EXACT` with `np.array_equal`, on a draw row and a round-1 row. +- [ ] Step 4: Commit `feat(rung4): the play-out takes its draws from one seam; exact is the seam returning everything, bit for bit`. + +### Task 3: the five samplers, each unbiased and non-vacuous + +**Files:** `boost_target.py` (`_Draws` bodies, the playouts loop, the per-row stream), `test_boost_target.py` + +- [ ] Step 1: Failing test `test_each_sampled_source_is_unbiased_and_actually_samples`, parametrized over `opponent_range=1`, `board_cards=1`, `opponent_draws=1`, `hero_draws=1`, `opponent_actions=1`, and `Sampling()` (all at once). Each case uses a row where that source is live, under `build(holes=WIDE)`, a random positive table, 400 seeds (t = seed). Assertions: + - for every action column, |mean − exact| < 4 · s.e. + - `assert_pooled_fair` on the pooled z + - the per-column spread is > 0 for at least one column (Review Focus 1) + - `arrival` equals `arrivals()` exactly +- [ ] Step 2: Implement the samplers: + - **Board:** `rng.permuted` over the 14 children per batch row, take the first k, scale 14/k into `u`. + - **Replacements:** k of 12 per (row, hole) or (row, hero state), via argsort of uniforms, scale 12/k. + - **Actions:** `rng.multinomial(k, pvals) / k`; rows of invalid states get uniform `pvals`, and their weight is already zero. + - **Range:** per entry, mask the states that clash with the hero's cards, draw k states ∝ `u·mask` with `rng.choice`, and put W/k on each draw (with multiplicity); W = 0 returns zeros. + - **Play-outs:** loop M times per row, averaging the numerator; divide once by the exact arrival. + + Run: PASS. +- [ ] Step 3: Failing test, then passing: `test_a_rows_sample_depends_on_seed_t_seat_row_only`. The same rows in reversed order, split over two calls, and with a duplicate give equal results; t + 1 gives a different result (Review Focus 5). +- [ ] Step 4: Failing test, then passing: `test_more_playouts_shrink_the_spread`. Across 200 seeds, the variance at `playouts=8` is within [1/16, 1/4] of the variance at `playouts=1`. +- [ ] Step 5: Failing test, then passing: `test_an_unreachable_row_is_zero_under_any_sampling`, with `opponent_range=1` (Review Focus 4). +- [ ] Step 6: Commit `feat(rung4): any source below a boosted row can be sampled k of n — each unbiased against the exact sum, on a per-row stream`. + +### Task 4: cost table, docstring, PR, and PR 2 + +- [ ] Step 1: Extend the benchmark test to print ms per row for `EXACT` vs `Sampling()` at round-1, draw and round-2 rows. Keep the existing round-2 ceiling, applied to `EXACT`. +- [ ] Step 2: Rewrite the module docstring: what is sampled, why the arrival stays exact (ratio bias), why k-of-n without replacement for chance and i.i.d. for σ, the stream, and the measured costs. Every number in it must be recomputed by a test or printed by the benchmark. +- [ ] Step 3: `uv run pytest`, the full default suite: PASS. Push to `AlecROndo/canary-handoff-pr-1-build`, retitle #89 and rewrite its body. +- [ ] Step 4: PR 2: in a temporary detached worktree on `origin/AlecROndo/pr-2-handoff-prompt`, `git merge origin/AlecROndo/canary-handoff-pr-1-build` (merge, never rebase), run `uv run pytest`, and push `HEAD:AlecROndo/pr-2-handoff-prompt`. Workspace `bariloche`'s local branch then needs a `git pull`; say so. +- [ ] Step 5: Drive #89 and #91 to merge-ready: checks green, `claude-review` findings dispositioned. diff --git a/src/drawmaha_solver/minidrawmaha/AGENTS.md b/src/drawmaha_solver/minidrawmaha/AGENTS.md index 4dbc0ec7..0250b052 100644 --- a/src/drawmaha_solver/minidrawmaha/AGENTS.md +++ b/src/drawmaha_solver/minidrawmaha/AGENTS.md @@ -14,10 +14,13 @@ Three subpackages, and one module that stays alone on purpose: sampling, one hand per seat), `lockstep` (W hands per seat against one frozen σ; the reference), `parallel` (the same in W processes over shared memory), `lockstep_run` (resume, grid checkpoints, progress), and - `exact_target` (rung 4's boost oracle: the regret a row's sampled targets - estimate, summed over every opponent hand and every physical entry of the - key, and the probability a walk arrives there; rules-only, and it must - never import `exploitability.py`, which grades what it feeds). + `boost_target` (rung 4's boost target: extra samples of a high-noise row's + regret, each source below the row — board card, both seats' replacements, + the opponent's actions and range — sampled k of n or enumerated as a + `Sampling` names, unbiased, with `EXACT`, the sum over every opponent hand + and every physical entry of the key, as their limit; and the exact + probability a walk arrives there; rules-only, and it must never import + `exploitability.py`, which grades what it feeds). - `answer/` — everything built on the solved game. `strategy` (the frozen LCFR average, a release asset pinned by SHA-256), `analysis` (`minidraw-analysis`, writes `figures/rung3/`), `play` (`minidraw-play`), diff --git a/src/drawmaha_solver/minidrawmaha/solver/exact_target.py b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py similarity index 72% rename from src/drawmaha_solver/minidrawmaha/solver/exact_target.py rename to src/drawmaha_solver/minidrawmaha/solver/boost_target.py index 23fb1b17..b5066ae6 100644 --- a/src/drawmaha_solver/minidrawmaha/solver/exact_target.py +++ b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py @@ -1,17 +1,45 @@ -"""The exact target: what the walk's regret sample at a table row is worth on average, computed. +"""The boost target: extra, lower-noise samples of a high-noise row's regret, unbiased, with the exact sum as their limit. Rung 4 learns every infoset from the regret samples the walks bank there, `r = u − ⟨σ, u⟩` with one opponent hand and one line of opponent play drawn per visit (`mccfr.traverse`, `lockstep.walk`). At a rare round-2 row a visit comes once in thousands of iterations and its sample swings by chips, almost -all of it the opponent's hidden hand. This module computes, for any table -row, the quantity those samples estimate — the regret averaged over every -hand the opponent could hold, weighted by how often the deck and the -opponent's own play bring exactly that hand there — and the probability that -one walk arrives at the row at all. A later stage (the "boost") adds one row -with that regret at the weight the walk's samples would have carried, so a -targeted infoset is learned from an exact number instead of a handful of -noisy ones. +all of it the opponent's hidden hand. This module draws more samples of that +same quantity at a chosen row, each with less noise than a walk's: it starts +at the row, faces the opponent's whole range (or k hands of it), and plays +the hand out with each source of randomness below — the board card, both +seats' replacements, the opponent's actions — sampled k of n or enumerated, +as a `Sampling` names. With every source enumerated (`EXACT`) the target is +the exact conditional mean the walk's samples estimate, and that is what the +tests pin the sampled forms to. A later stage (the "boost") banks the target +at the weight the walk's samples would have carried there, so a targeted +infoset is learned from a few good samples instead of a handful of noisy +ones. + +**What is sampled, and why each stays unbiased.** Every sampled source +replaces a sum over n branches by k of them, so that the expectation over +the draw is the sum: + +- a chance fan (the second board card, 14 branches; a throw's replacement, + 12 per hand, for either seat) keeps k of its n branches without + replacement, each scaled n/k: each branch is kept with probability k/n; +- the opponent's actions send each opponent state's weight down k actions + drawn from its σ at 1/k each, instead of splitting it by σ: the expected + share of an action is σ(a); +- the opponent range draws k states per physical entry in proportion to + their weight, among those sharing no card with the entry's own state, each + carrying 1/k of the entry's total; +- `playouts` independent play-outs are averaged. + +The traverser's own actions are never sampled, as the walk never samples +them: the regret is defined against each of them. The arrival — the ratio's +denominator below — is always exact. It is the cheap half (the opponent's +reach, back-solved upward, never sampled), and an exact denominator under an +unbiased numerator keeps the ratio unbiased; a sampled one would not, since +the mean of a ratio is not the ratio of the means. Each row draws from +`default_rng([seed, t, seat, row, "BOOST"])`, its own stream, independent of +the walk's, so a row's target depends on (seed, t, seat, row) and never on +which rows share the call. **What is computed.** For a row standing for the key I of seat `seat`: @@ -90,26 +118,31 @@ world-by-world walk through `MiniState` and the mean of thousands of real walks. -**Cost, measured on this Mac (2026-11-05, a random table).** The description -is built once per process, 5–7 s — the key maps, the draw maps, the -public tree with parent links, the showdown ranks — and holds no showdown -matrix: shares and compatibility are computed per terminal from the holes' -ranks and card sets, folded over the opponent's 455 holes. `sigma` over the -whole table takes 0.12 s. A round-2 row costs 0.8 ms (1.5 ms when 2,240 rows -are spread over every round-2 point, where the reach memo warms less per -row), and the arrival alone 0.3 ms. A draw row costs 45–85 ms and a round-1 -row 0.2–0.6 s, because the value below them expands over the traverser's own -replacements (36 states per throw) and the second board card (14 boards): -the opponent's vector is carried once per board and shared by every hero -state on it, which is what keeps those within a second. The opponent's reach -is memoised per call at every node with one board card out that lies on -the path to a decision (66: the 29 level-1 decision points and 37 chance -nodes, 15 boards each) and at every round-2 opening decision (20: one per -round-1 line and pair of draw counts, 210 boards each), 58 KB a (node, -board) pair, so the memo is at most 66 × 15 + 20 × 210 = 5,190 vectors, -302 MB — a count `tests/minidrawmaha/test_exact_target.py` re-derives -from the tree; a call over 2,240 round-2 rows peaked at about 260 MB. `tests/minidrawmaha/test_exact_target.py` prints the -round-2 figure on every run. +**Cost, measured on this Mac (2026-11-06, a table with every action +positive, so nothing below a row prunes).** The description is built once +per process, 5–7 s — the key maps, the draw maps, the public tree with +parent links, the showdown ranks — and holds no showdown matrix: shares and +compatibility are computed per terminal from the holes' ranks and card sets, +folded over the opponent's 455 holes. `sigma` over the whole table takes +0.12 s. Exactly, a round-2 row costs about 2 ms, a draw row 160–230 ms and a +round-1 row 0.5–0.75 s, because the value below the last two expands over +the traverser's own replacements (36 states per throw) and the second board +card (14 boards). Under the walk's own sampling (`Sampling()`) a draw row +costs about 25–32 ms and a round-1 row about 100–130 ms: the board card and +the hero's replacements are what sampling cuts. A round-2 row costs more +sampled than exact, about 3–4 ms: against a whole range the opponent's +actions are split by σ in one array operation, and drawing them per state +only adds a draw and noise — action sampling earns its keep when +`opponent_range` is a few hands, where most lines carry no weight and are +skipped. `tests/minidrawmaha/test_boost_target.py` prints the three costs on +every run. The opponent's reach is memoised per call at every node with one +board card out that lies on the path to a decision (66: the 29 level-1 +decision points and 37 chance nodes, 15 boards each) and at every round-2 +opening decision (20: one per round-1 line and pair of draw counts, 210 +boards each), 58 KB a (node, board) pair, so the memo is at most +66 × 15 + 20 × 210 = 5,190 vectors, 302 MB — a count the test file +re-derives from the tree; a call over 2,240 round-2 rows peaked at about +260 MB. """ from __future__ import annotations @@ -148,6 +181,109 @@ # The widest ledger: the draw's four throws. Regret rows are padded to it. TARGET_WIDTH = max(len(DRAW_ACTIONS), len(BETTING_ACTIONS)) +# The fifth word of a boost stream, ASCII "BOOST": a row's stream can never +# equal a walk's four-word stream or the reservoir's "RESV" one. +_STREAM_TAG = int.from_bytes(b"BOOST", "big") + +# --------------------------------------------------------------------------- +# What the play-out below a boosted row samples +# --------------------------------------------------------------------------- + +_SOURCES = ("opponent_range", "board_cards", "opponent_draws", "hero_draws", "opponent_actions") + +@dataclass(frozen=True, slots=True) +class Sampling: + """How many of each random thing below a boosted row one play-out takes; `None` takes all of them. + + - `playouts` — independent play-outs from the row, averaged. + - `opponent_range` — `None` plays against the whole range; k draws k + opponent hands per physical entry, with replacement, in proportion to + their weight, each carrying 1/k of the entry's total. + - `board_cards` — k of the 14 cards the second board card can be, + without replacement, each scaled 14/k. + - `opponent_draws`, `hero_draws` — k of the 12 replacements a throw can + bring, per hand, without replacement, each scaled 12/k. + - `opponent_actions` — `None` splits every opponent hand's weight over its + actions by σ; k sends it down k actions drawn from σ, 1/k each. + + A chance count at or above its fan is that source enumerated. The + defaults are the walk's own sampling below the row against the whole + opponent range; `EXACT` enumerates everything and is the oracle. The + traverser's own actions are never sampled — the walk enumerates them, and + the regret is defined against them — only the cards its throws bring. + """ + + playouts: int = 1 + opponent_range: int | None = None + board_cards: int | None = 1 + opponent_draws: int | None = 1 + hero_draws: int | None = 1 + opponent_actions: int | None = 1 + + def __post_init__(self) -> None: + for name in ("playouts", *_SOURCES): + value = getattr(self, name) + if value is None and name != "playouts": + continue + if isinstance(value, bool) or not isinstance(value, int) or value < 1: + allowed = "a positive int" if name == "playouts" else "a positive int, or None for all" + raise ValueError(f"Sampling.{name} must be {allowed}, got {value!r}") + if self.is_exact and self.playouts != 1: + raise ValueError( + f"Sampling enumerates every source, so every play-out is the same; playouts must be 1, got {self.playouts}" + ) + + @property + def is_exact(self) -> bool: + return all(getattr(self, name) is None for name in _SOURCES) + +EXACT = Sampling( + playouts=1, opponent_range=None, board_cards=None, opponent_draws=None, hero_draws=None, opponent_actions=None +) + +@dataclass(frozen=True, slots=True, eq=False) +class _Draws: + """The play-out's one source of randomness: under `EXACT` every method hands back everything and the RNG is never touched.""" + + sampling: Sampling + rng: np.random.Generator | None + + def subset(self, k: int | None, n: int, shape: tuple[int, ...]) -> np.ndarray | None: + """k of n positions for every index of `shape`, without replacement; `None` for all n.""" + if k is None or k >= n: + return None + return np.argsort(self.rng.random((*shape, n)), axis=-1)[..., :k] + + def actions(self, probs: np.ndarray) -> np.ndarray: + """Per opponent state, the share of its weight each action carries: σ itself, or k draws from σ at 1/k each.""" + k = self.sampling.opponent_actions + if k is None: + return probs + cdf = np.cumsum(probs, axis=-1) + cdf[..., -1] = 1.0 # a row summing to 1 − ulp must still pick an action + columns = np.arange(probs.shape[-1]) + counts = np.zeros_like(probs) + for _ in range(k): + pick = (self.rng.random(probs.shape[:-1])[..., None] < cdf).argmax(axis=-1) + counts += pick[..., None] == columns + return counts / k + + def opponents(self, u: np.ndarray, clash: np.ndarray) -> np.ndarray: + """Per entry, the opponent's weighted vector, or k hands drawn from its live weight at 1/k of the total each.""" + k = self.sampling.opponent_range + if k is None: + return u + live = np.where(clash, 0.0, u) + totals = live.sum(axis=1) + out = np.zeros_like(u) + for e, total in enumerate(totals): + if total > 0.0: + picks = self.rng.choice(u.shape[1], size=k, p=live[e] / total) + out[e] = np.bincount(picks, minlength=u.shape[1]) * (total / k) + return out + +_EXACT_DRAWS = _Draws(EXACT, None) + # --------------------------------------------------------------------------- # The frame: physical states and board rows # --------------------------------------------------------------------------- @@ -556,18 +692,19 @@ def _own_deal(weight: np.ndarray) -> dict[tuple[int, int], np.ndarray]: return {(1, 0): weight, (2, 0): weight, (2, 1): post} # --------------------------------------------------------------------------- -# The exact targets +# The targets # --------------------------------------------------------------------------- _Sigma = Sequence[np.ndarray] @dataclass(frozen=True, eq=False) -class ExactTargets: - """Exact regret and arrival for any table row, under a strategy read from the materialised table. +class BoostTargets: + """Sampled (or exact) regret and exact arrival for any table row, under a strategy read from the materialised table. - exact = ExactTargets.build() # once per process - sigma = exact.sigma(table) # the walk's σ, per point - regret, arrival = exact.targets(sigma, seat=0, rows=rows) + boost = BoostTargets.build() # once per process + sigma = boost.sigma(table) # the walk's σ, per point + regret, arrival = boost.targets(sigma, seat=0, rows=rows, sampling=Sampling(), seed=seed, t=t) + regret, arrival = boost.targets(sigma, seat=0, rows=rows, sampling=EXACT) # the oracle `regret` is `(n, 4)` float64, zero past each row's width; `arrival` is `(n,)`, the expected number of visits one walk with traverser `seat` @@ -584,7 +721,7 @@ class ExactTargets: own_deal: tuple[dict[tuple[int, int], np.ndarray], dict[tuple[int, int], np.ndarray]] @classmethod - def build(cls, holes: tuple[Iterable[Hole], Iterable[Hole]] | None = None) -> ExactTargets: + def build(cls, holes: tuple[Iterable[Hole], Iterable[Hole]] | None = None) -> BoostTargets: """The description (shared, built once) with this deal set's weights. The default is cached.""" if holes is None: return _full_game() @@ -638,14 +775,33 @@ def sigma(self, table: PackedTable) -> tuple[np.ndarray, ...]: # -- the targets -------------------------------------------------------- def targets( - self, sigma: _Sigma, *, seat: int, rows: np.ndarray + self, + sigma: _Sigma, + *, + seat: int, + rows: np.ndarray, + sampling: Sampling, + seed: int | None = None, + t: int | None = None, ) -> tuple[np.ndarray, np.ndarray]: - """Exact regret `(n, 4)` and arrival `(n,)` — expected visits per walk — for table `rows`, all of them `seat`'s, under `sigma`. - - A row the walk cannot reach (zero arrival) gets zero regret, not NaN; - a row of the other seat raises, as does one outside the table. + """Regret `(n, 4)` and arrival `(n,)` — expected visits per walk — for table `rows`, all of them `seat`'s, under `sigma`. + + The regret is exact under `EXACT` and an unbiased estimate of it + otherwise; the arrival is exact under any `sampling`. A sampled call + names `seed` and `t`, and row r draws from + `default_rng([seed, t, seat, r, "BOOST"])` alone, so its target does + not depend on which rows share the call or in what order. A row the + walk cannot reach (zero arrival) gets zero regret, not NaN; a row of + the other seat raises, as does one outside the table. """ + if not isinstance(sampling, Sampling): + raise TypeError(f"sampling must be a Sampling (EXACT for the exact sum), got {sampling!r}") + if not sampling.is_exact: + for name, value in (("seed", seed), ("t", t)): + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError(f"a sampled target needs a non-negative int {name}, got {value!r}") rows = self._rows(seat, rows) + d = self.description regret = np.zeros((rows.size, TARGET_WIDTH)) arrival = np.zeros(rows.size) memo: dict[tuple[int, int, int], np.ndarray] = {} @@ -655,10 +811,25 @@ def targets( arrival[i] = total if total <= 0.0: continue - values = self._action_values(sigma, seat, node, u, states[:, None], boards)[:, 0, :] + draws = ( + _EXACT_DRAWS + if sampling.is_exact + else _Draws(sampling, np.random.default_rng([seed, t, seat, int(rows[i]), _STREAM_TAG])) + ) + # Opponent states sharing a card with the entry's hero state; only + # a range draw needs them, since the play-out's compat masks them. + clash = ( + None + if sampling.opponent_range is None + else (d.holds.astype(np.uint8) @ d.holds[states].T.astype(np.uint8)).T > 0 + ) mix = sigma[node.point][position] - baseline = values @ mix - regret[i, : len(mix)] = (values - baseline[:, None]).sum(axis=0) / total + summed = np.zeros(len(mix)) + for _ in range(sampling.playouts): + facing = u if clash is None else draws.opponents(u, clash) + values = self._action_values(sigma, seat, node, facing, states[:, None], boards, draws)[:, 0, :] + summed += (values - (values @ mix)[:, None]).sum(axis=0) + regret[i, : len(mix)] = summed / (sampling.playouts * total) return regret, arrival def arrivals(self, sigma: _Sigma, *, seat: int, rows: np.ndarray) -> np.ndarray: @@ -761,7 +932,7 @@ def _reach(self, sigma: _Sigma, carried: int, node: _Node, b: int, memo: dict) - # done once per board, not once per hero continuation. def _action_values( - self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray + self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray, draws: _Draws ) -> np.ndarray: """Hero's weighted value below each action of `node`: `(nb, nh, width)` for boards b, hero states h (nb, nh), opponent vectors u (nb, S).""" d = self.description @@ -769,27 +940,36 @@ def _action_values( values = np.zeros((nb, nh, len(node.actions))) for edge in node.edges: if all(draw is None for draw in edge.draws): - below = self._value(sigma, hero, edge.child, u, h, b) + below = self._value(sigma, hero, edge.child, u, h, b, draws) for column in edge.actions: values[..., column] = below continue # The throw edge: the three throws' replacement states, stacked, # walk the one subtree below in one pass and are split after. + # Under sampling each (board, hero state) keeps k of its twelve + # replacements per throw, scaled 12/k; hero's throws themselves + # are all walked. hole = d.hole_of[h] fan = edge.draws[0].targets.shape[-1] - stacked = np.concatenate( - [draw.targets[b[:, None], hole] for draw in edge.draws], axis=-1 - ) # (nb, nh, 3 × 12) - below = self._value(sigma, hero, edge.child, u, stacked.reshape(nb, -1), b) - below = below.reshape(nb, nh, len(edge.draws), fan).sum(axis=-1) + parts = [] + for draw in edge.draws: + replacements = draw.targets[b[:, None], hole] # (nb, nh, 12) + pick = draws.subset(draws.sampling.hero_draws, fan, (nb, nh)) + parts.append(replacements if pick is None else np.take_along_axis(replacements, pick, axis=-1)) + kept = parts[0].shape[-1] + stacked = np.concatenate(parts, axis=-1) # (nb, nh, 3 × kept) + below = self._value(sigma, hero, edge.child, u, stacked.reshape(nb, -1), b, draws) + below = below.reshape(nb, nh, len(edge.draws), kept).sum(axis=-1) + if kept < fan: + below *= fan / kept for j, column in enumerate(edge.actions): values[..., column] = below[..., j] return values def _value( - self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray + self, sigma: _Sigma, hero: int, node: _Node, u: np.ndarray, h: np.ndarray, b: np.ndarray, draws: _Draws ) -> np.ndarray: - """Hero's weighted expected chips from `node`, `(nb, nh)`, both seats at σ, the deck averaged.""" + """Hero's weighted expected chips from `node`, `(nb, nh)`, both seats at σ, the deck averaged (or sampled, by `draws`).""" d = self.description nb, nh = h.shape if node.kind is NodeKind.TERMINAL: @@ -802,33 +982,46 @@ def _value( if node.deals_board: level = d.levels[node.child.level] kids = level.children_of[b] # (nb, 14) + full = kids.shape[1] + pick = draws.subset(draws.sampling.board_cards, full, (nb,)) + if pick is not None: + kids = np.take_along_axis(kids, pick, axis=1) fan = kids.shape[1] spread = node.prob * u[:, None, :] * level.ok[kids] + if fan < full: + spread *= full / fan below = self._value( sigma, hero, node.child, - spread.reshape(-1, N_STATES), np.repeat(h, fan, axis=0), kids.ravel(), + spread.reshape(-1, N_STATES), np.repeat(h, fan, axis=0), kids.ravel(), draws, ) return below.reshape(nb, fan, nh).sum(axis=1) - return self._value(sigma, hero, node.child, node.prob * u * d.levels[node.level].ok[b], h, b) + return self._value( + sigma, hero, node.child, node.prob * u * d.levels[node.level].ok[b], h, b, draws + ) if node.player == hero: index = d.keys[(node.level, node.discards[hero])].index[b[:, None], h] # (nb, nh) valid = index >= 0 mix = np.where(valid[..., None], sigma[node.point][np.where(valid, index, 0)], 0.0) - return (self._action_values(sigma, hero, node, u, h, b) * mix).sum(axis=-1) + return (self._action_values(sigma, hero, node, u, h, b, draws) * mix).sum(axis=-1) index = d.keys[(node.level, node.discards[node.player])].index[b] # (nb, S) valid = index >= 0 safe = np.where(valid, index, 0) - table = sigma[node.point] + share = draws.actions(sigma[node.point][safe]) # (nb, S, width) total = np.zeros((nb, nh)) for edge in node.edges: pushed = np.zeros_like(u) for column, draw in zip(edge.actions, edge.draws, strict=True): - played = u * np.where(valid, table[safe, column], 0.0) + played = u * np.where(valid, share[..., column], 0.0) if draw is None: pushed += played - else: - pushed += _push(played, draw, b) - total += self._value(sigma, hero, edge.child, pushed, h, b) + continue + pick = draws.subset(draws.sampling.opponent_draws, draw.targets.shape[-1], (nb, len(HOLES))) + pushed += _push(played, draw, b, pick) + # A line no opponent state takes is worth nothing below; under + # action sampling most are, and walking them would cost the + # whole subtree for a zero. + if pushed.any(): + total += self._value(sigma, hero, edge.child, pushed, h, b, draws) return total # -- the showdown and compatibility ------------------------------------ @@ -866,17 +1059,22 @@ def _showdown( share = 0.25 * (2.0 + inner + outer) return compat, (terms * share).sum(axis=1) * live -def _push(played: np.ndarray, draw: _DrawMap, b: np.ndarray) -> np.ndarray: - """Each pair's vector through a throw on its level-1 board: every pre-draw hole spreads over its twelve replacements.""" +def _push(played: np.ndarray, draw: _DrawMap, b: np.ndarray, pick: np.ndarray | None = None) -> np.ndarray: + """Each pair's vector through a throw on its level-1 board: every pre-draw hole spreads over its twelve replacements, or over the k of them `pick` keeps, scaled 12/k.""" m = played.shape[0] targets = draw.targets[b] # (m, 455, 12) + full = targets.shape[2] + if pick is not None: + targets = np.take_along_axis(targets, pick, axis=2) offsets = (np.arange(m) * N_STATES)[:, None, None] weights = np.repeat(played[:, draw.source], targets.shape[2], axis=1).ravel() + if targets.shape[2] < full: + weights *= full / targets.shape[2] return np.bincount((offsets + targets).ravel(), weights=weights, minlength=m * N_STATES).reshape( m, N_STATES ) @cache -def _full_game() -> ExactTargets: +def _full_game() -> BoostTargets: weights = _weights(None) - return ExactTargets(_description(), weights, (_own_deal(weights[0]), _own_deal(weights[1]))) + return BoostTargets(_description(), weights, (_own_deal(weights[0]), _own_deal(weights[1]))) diff --git a/tests/minidrawmaha/test_exact_target.py b/tests/minidrawmaha/test_boost_target.py similarity index 74% rename from tests/minidrawmaha/test_exact_target.py rename to tests/minidrawmaha/test_boost_target.py index 4a81c200..23e0e72e 100644 --- a/tests/minidrawmaha/test_exact_target.py +++ b/tests/minidrawmaha/test_boost_target.py @@ -1,8 +1,10 @@ -"""The exact target: what one sampled regret row is worth on average, computed instead of sampled. +"""The boost target: extra samples of a row's regret, each unbiased for the exact sum the tests first pin down. -The module under test is rung 4's boost oracle: for a table row it sums, over -every physical entry of the key and every opponent hand, the regret the walk -would bank there, weighted by how often the walk arrives. The claims, in +The module under test is rung 4's boost target. Exactly (`EXACT`), for a +table row it sums, over every physical entry of the key and every opponent +hand, the regret the walk would bank there, weighted by how often the walk +arrives; sampled, it estimates that sum from play-outs that keep k of n of +each source below the row. The claims, in order: it imports the rules and never the grader; `sigma` is the ledger's own σ row by row; the back-solved opponent reach equals the grader's carried vector on every board of every public point, in the full game and in a @@ -13,7 +15,14 @@ restricted game they match the mean and the visit count of thousands of `Collector` walks, row by row within four standard errors and pooled (tests b and c); rows of the wrong seat are refused and an unreachable row is zero; -and the cost per round-2 row is recorded (test d, a benchmark). +and the cost per round-2 row is recorded (test d, a benchmark). Then the +sampled forms (e): counts that are not positive ints are refused by name; a +count at or above its fan is enumeration, bit for bit; each source alone, +and all of them together, is unbiased against the exact sum at a row where +it is live, and actually moves the target there; a row's sample depends on +(seed, t, seat, row) only; play-outs shrink the spread as 1/M; an +unreachable row is zero under any sampling; and sampling a draw or round-1 +row costs a fraction of its exact sum. Helpers are restated rather than imported from other test modules, as the suite's rule has it. @@ -46,18 +55,20 @@ canonical_picture, random_deal, ) -from drawmaha_solver.minidrawmaha.solver.exact_target import ( +from drawmaha_solver.minidrawmaha.solver.boost_target import ( HOLES, N_STATES, SLOTS, - ExactTargets, + EXACT, + BoostTargets, + Sampling, board_row, state_index, ) from drawmaha_solver.minidrawmaha.solver.packed_table import PackedTable REPO = Path(__file__).parents[2] -MODULE = REPO / "src" / "drawmaha_solver" / "minidrawmaha" / "solver" / "exact_target.py" +MODULE = REPO / "src" / "drawmaha_solver" / "minidrawmaha" / "solver" / "boost_target.py" FULL_TARGET = os.environ.get("MINIDRAWMAHA_FULL_TARGET") == "1" # --------------------------------------------------------------------------- @@ -326,8 +337,8 @@ def test_the_module_imports_the_rules_and_never_the_grader_or_the_net(): # --------------------------------------------------------------------------- def test_build_is_one_object_per_deal_set_over_a_shared_description(): - full = ExactTargets.build() - assert ExactTargets.build() is full # the default is cached: the description takes seconds to build + full = BoostTargets.build() + assert BoostTargets.build() is full # the default is cached: the description takes seconds to build assert len(full.points) == 141 # The reach memo's bound, as the module states it: every node with one # board card out on the path from a decision to the root, and every @@ -367,7 +378,7 @@ def test_build_is_one_object_per_deal_set_over_a_shared_description(): assert sizes <= {1, 2, 3, 6} def test_sigma_is_the_ledgers_strategy_row_by_row(): - full = ExactTargets.build() + full = BoostTargets.build() table = random_table(1) sigma = full.sigma(table) assert len(sigma) == 141 @@ -392,7 +403,7 @@ def test_sigma_is_the_ledgers_strategy_row_by_row(): @pytest.mark.parametrize("holes", [None, RESTRICTED], ids=["full", "restricted"]) @pytest.mark.parametrize("carried", [0, 1]) def test_reach_equals_the_graders_carried_reach_on_every_board(holes, carried): - exact = ExactTargets.build(holes=holes) + exact = BoostTargets.build(holes=holes) game = grader.minidrawmaha_game(holes=holes) table = random_table(2) sigma = exact.sigma(table) @@ -417,7 +428,7 @@ def test_arrival_at_a_seats_first_decision_is_the_graders_frequency_and_sums_to_ # Before a seat has acted, own reach is the deal alone, so the grader's # `frequency` (own × compat) at its opening points is exactly the walk's # arrival; and every walk reaches one of them. - exact = ExactTargets.build() + exact = BoostTargets.build() game = grader.minidrawmaha_game() table = random_table(3) sigma = exact.sigma(table) @@ -445,7 +456,7 @@ def test_arrival_at_a_seats_first_decision_is_the_graders_frequency_and_sums_to_ # The full computation agrees with the arrival-only path (a round-1 # row's value walks the whole game below it, so a handful of rows). some = rows[:: max(1, len(rows) // 4)] - regret, with_values = exact.targets(sigma, seat=seat, rows=some) + regret, with_values = exact.targets(sigma, seat=seat, rows=some, sampling=EXACT) assert regret.shape == (len(some), 4) and np.isfinite(regret).all() np.testing.assert_array_equal(with_values, arrival[:: max(1, len(rows) // 4)]) assert total == pytest.approx(1.0, abs=1e-12) @@ -486,12 +497,12 @@ def canonical_key(private): @pytest.mark.parametrize("seat", [0, 1]) def test_regret_and_arrival_equal_a_world_by_world_expectation(seat): pairs = allowed_pairs(*SMALL) - exact = ExactTargets.build(holes=SMALL) + exact = BoostTargets.build(holes=SMALL) table = positive_table(4) sigma = exact.sigma(table) keys = small_game_keys(seat) rows = rows_at(table, keys) - regret, arrival = exact.targets(sigma, seat=seat, rows=rows) + regret, arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=EXACT) for key, row_regret, row_arrival in zip(keys, regret, arrival, strict=True): expected, reach = world_by_world(table, key, pairs) assert reach > 0, str(key) @@ -525,7 +536,7 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): # (swapped reach, a dropped entry, the wrong chance factor) far above. walks = 6_000 _ALLOWED[:] = allowed_pairs(*WIDE) - exact = ExactTargets.build(holes=WIDE) + exact = BoostTargets.build(holes=WIDE) table = random_table(5) sigma = exact.sigma(table) with Collector(seed=11, workers=0, deal=restricted_deal) as collector: @@ -534,7 +545,7 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): rows, counts = np.unique(offered.advantage_rows, return_counts=True) enough = counts >= 40 rows, counts = rows[enough], counts[enough] - regret, arrival = exact.targets(sigma, seat=0, rows=rows) + regret, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) # Rows are kept by their EXPECTED count, 80 or more, so the observed # cutoff of 40 sits 4.5 standard errors below every kept row's mean and # the selection cannot bias z: a row chosen by its observed count alone @@ -567,7 +578,7 @@ def test_regret_and_arrival_match_the_sampled_walks_row_by_row_and_pooled(): @pytest.mark.parametrize("seat", [0, 1]) def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat): walks = 200_000 - exact = ExactTargets.build() + exact = BoostTargets.build() table = random_table(6) sigma = exact.sigma(table) with Collector(seed=12, workers=8, deal=random_deal) as collector: @@ -602,24 +613,24 @@ def test_arrival_is_the_visit_rate_of_two_hundred_thousand_whole_game_walks(seat # --------------------------------------------------------------------------- def test_rows_of_the_other_seat_and_outside_the_table_are_refused(): - exact = ExactTargets.build() + exact = BoostTargets.build() table = random_table(7) sigma = exact.sigma(table) points = public_decision_points() seat1 = next(i for i, point in enumerate(points) if point.player == 1) row_of_seat_1 = table.row_of(key_on(points[seat1], shape_of(points[seat1])[0])) with pytest.raises(ValueError, match="seat"): - exact.targets(sigma, seat=0, rows=np.array([row_of_seat_1])) + exact.targets(sigma, seat=0, rows=np.array([row_of_seat_1]), sampling=EXACT) with pytest.raises(ValueError, match="seat"): - exact.targets(sigma, seat=2, rows=np.array([0])) + exact.targets(sigma, seat=2, rows=np.array([0]), sampling=EXACT) for bad in (-1, len(table)): with pytest.raises(IndexError): - exact.targets(sigma, seat=0, rows=np.array([bad])) + exact.targets(sigma, seat=0, rows=np.array([bad]), sampling=EXACT) def test_an_unreachable_row_has_zero_arrival_and_zero_regret(): # Seat 1 never bets or raises: every seat-0 round-1 row facing a bet is # unreachable, and the exact target says so with zeros rather than NaN. - exact = ExactTargets.build() + exact = BoostTargets.build() table = random_table(8) points = public_decision_points() for p, point in enumerate(points): @@ -637,7 +648,149 @@ def test_an_unreachable_row_has_zero_arrival_and_zero_regret(): and point.betting[0][-1:] == (Action.POT,) ) rows = np.arange(len(shape_of(points[facing]))) + table.row_of(key_on(points[facing], shape_of(points[facing])[0])) - regret, arrival = exact.targets(sigma, seat=0, rows=rows) + regret, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + assert not arrival.any() and not regret.any() + assert np.isfinite(regret).all() + +# --------------------------------------------------------------------------- +# (e) sampling below the row: unbiased against the exact sum, seeded per row +# --------------------------------------------------------------------------- + +def only(**sampled) -> Sampling: + """A Sampling that enumerates every source but the ones named.""" + return Sampling(**{ + "opponent_range": None, "board_cards": None, "opponent_draws": None, + "hero_draws": None, "opponent_actions": None, **sampled, + }) + +def row_on(table: PackedTable, point: int, position: int = 7) -> np.ndarray: + points = public_decision_points() + return np.array([table.row_of(key_on(points[point], shape_of(points[point])[position]))]) + +def sampled(exact, sigma, seat, rows, sampling, seeds) -> np.ndarray: + """One sampled target per seed, `(len(seeds), 4)`, t equal to the seed.""" + return np.stack([ + exact.targets(sigma, seat=seat, rows=rows, sampling=sampling, seed=0, t=s)[0][0] for s in seeds + ]) + +@pytest.mark.parametrize( + "bad", + [dict(board_cards=0), dict(hero_draws=-1), dict(opponent_actions=True), dict(opponent_range=1.5), dict(playouts=0)], + ids=lambda d: next(iter(d)), +) +def test_sampling_refuses_counts_that_are_not_positive_ints(bad): + field = next(iter(bad)) + with pytest.raises(ValueError, match=f"Sampling.{field}"): + Sampling(**bad) + +def test_exact_with_more_than_one_playout_is_refused(): + with pytest.raises(ValueError, match="playouts must be 1"): + only(playouts=2) + +def test_a_sampled_call_names_its_seed_and_t(): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, 4) + with pytest.raises(ValueError, match="seed"): + exact.targets(sigma, seat=0, rows=rows, sampling=Sampling(), t=0) + with pytest.raises(ValueError, match="t,|non-negative int t"): + exact.targets(sigma, seat=0, rows=rows, sampling=Sampling(), seed=0) + with pytest.raises(TypeError, match="Sampling"): + exact.targets(sigma, seat=0, rows=rows, sampling=None) + +@pytest.mark.parametrize("point", [2, 0], ids=["draw row", "round-1 row"]) +def test_a_count_at_or_above_its_fan_is_enumeration_bit_for_bit(point): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, point) + full = only(board_cards=14, opponent_draws=12, hero_draws=99) + want = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + got = exact.targets(sigma, seat=0, rows=rows, sampling=full, seed=0, t=0) + assert np.array_equal(want[0], got[0]) and np.array_equal(want[1], got[1]) + +# Each source alone at a row where it is live, then all of them at once. The +# draw-row cases walk the whole draw and round 2 below them, about 0.2 s a +# call when nothing prunes, so they take fewer seeds. +UNBIASED = [ + ("opponent_range", only(opponent_range=1), 4, 0, 400), + ("opponent_actions", only(opponent_actions=1), 4, 0, 400), + ("board_cards", only(board_cards=1), 2, 0, 200), + ("hero_draws, seat 0", only(hero_draws=1), 2, 0, 60), + ("hero_draws, seat 1", only(hero_draws=1), 3, 1, 60), + ("opponent_draws", only(opponent_draws=1), 2, 0, 60), + ("the walk's sampling, whole range", Sampling(), 2, 0, 200), + ("everything at one", Sampling(opponent_range=1), 2, 0, 200), + ("two of each, two play-outs", Sampling(playouts=2, opponent_range=2, board_cards=2, opponent_draws=2, + hero_draws=2, opponent_actions=2), 2, 0, 100), +] + +@pytest.mark.parametrize(("sampling", "point", "seat", "n"), [c[1:] for c in UNBIASED], ids=[c[0] for c in UNBIASED]) +def test_each_sampled_source_is_unbiased_and_actually_samples(sampling, point, seat, n): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, point) + want, arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=EXACT) + assert arrival[0] > 0.0 + draws = sampled(exact, sigma, seat, rows, sampling, range(n)) + spread = draws.std(axis=0, ddof=1) + live = spread > 0.0 + # Not vacuous: the source really varied the target at this row. + assert live.any(), "the sampled source never moved the target here" + z = (draws.mean(axis=0)[live] - want[0][live]) / (spread[live] / np.sqrt(n)) + assert (np.abs(z) < 4.5).all(), z + # A column the sample never moves is the exact one, bit for bit or near. + np.testing.assert_allclose(draws[:, ~live], np.broadcast_to(want[0][~live], draws[:, ~live].shape), atol=1e-12) + _, sampled_arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=sampling, seed=0, t=0) + assert sampled_arrival[0] == arrival[0] == exact.arrivals(sigma, seat=seat, rows=rows)[0] + +def test_a_rows_sample_depends_on_seed_t_seat_and_row_only(): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = np.concatenate([row_on(table, 4, 7), row_on(table, 2, 11), row_on(table, 4, 30)]) + at = lambda r, t=5: exact.targets(sigma, seat=0, rows=r, sampling=Sampling(opponent_range=1), seed=1, t=t)[0] + together = at(rows) + assert np.array_equal(at(rows[::-1]), together[::-1]) + assert np.array_equal(np.concatenate([at(rows[:1]), at(rows[1:])]), together) + assert np.array_equal(at(rows[[0, 0, 1, 2]]), together[[0, 0, 1, 2]]) + assert not np.array_equal(at(rows, t=6), together) + +def test_more_playouts_shrink_the_spread_as_one_over_m(): + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + rows = row_on(table, 4) + one = sampled(exact, sigma, 0, rows, only(opponent_actions=1), range(200)).var(axis=0, ddof=1) + eight = sampled(exact, sigma, 0, rows, only(opponent_actions=1, playouts=8), range(200)).var(axis=0, ddof=1) + live = one > 0.0 + ratio = eight[live] / one[live] + assert ((1 / 16 < ratio) & (ratio < 1 / 4)).all(), ratio + +def test_an_unreachable_row_is_zero_under_any_sampling(): + # The same construction as the exact test above: seat 1 never bets, so a + # seat-0 round-1 row facing a bet has no weight to draw a range from. + exact = BoostTargets.build() + table = random_table(8) + points = public_decision_points() + for p, point in enumerate(points): + if point.player == 1 and not point.is_draw_decision and Action.POT in point.actions: + column = point.actions.index(Action.POT) + for position in range(len(shape_of(point))): + _, start, width = table.slot_of(key_on(point, shape_of(point)[position])) + table.cumulative_regret[start + column] = -1.0 + if not (table.cumulative_regret[start : start + width] > 0).any(): + table.cumulative_regret[start + point.actions.index(Action.CHECK_CALL)] = 1.0 + sigma = exact.sigma(table) + facing = next( + i for i, point in enumerate(points) + if point.player == 0 and point.board_cards == 1 and not point.is_draw_decision + and point.betting[0][-1:] == (Action.POT,) + ) + rows = np.arange(4) + table.row_of(key_on(points[facing], shape_of(points[facing])[0])) + regret, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=Sampling(opponent_range=1), seed=0, t=0) assert not arrival.any() and not regret.any() assert np.isfinite(regret).all() @@ -646,7 +799,7 @@ def test_an_unreachable_row_has_zero_arrival_and_zero_regret(): # --------------------------------------------------------------------------- def test_a_round_two_row_costs_a_few_milliseconds(): - exact = ExactTargets.build() + exact = BoostTargets.build() table = random_table(9) sigma = exact.sigma(table) rng = np.random.default_rng(0) @@ -656,9 +809,9 @@ def test_a_round_two_row_costs_a_few_milliseconds(): rows = np.concatenate([ offsets[p] + rng.choice(len(shape_of(points[p])), size=10, replace=False) for p in round_two ]) - exact.targets(sigma, seat=0, rows=rows[:5]) # warm the description + exact.targets(sigma, seat=0, rows=rows[:5], sampling=EXACT) # warm the description started = time.perf_counter() - exact.targets(sigma, seat=0, rows=rows) + exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) per_row = (time.perf_counter() - started) / rows.size print(f"\nexact target: {per_row * 1e3:.2f} ms per round-2 row over {rows.size} rows") # The plan's figure is 1 ms a row and the measured one 0.8 ms. The @@ -667,3 +820,25 @@ def test_a_round_two_row_costs_a_few_milliseconds(): # cross it, narrow enough to catch a regression to the grader's # whole-board update, which is about a second a row. assert per_row < 0.050 + +def test_sampling_below_a_draw_or_round_one_row_costs_a_fraction_of_the_exact_sum(): + # The board card's fourteen branches and the hero's thirty-six + # replacements are what an exact draw or round-1 row pays for; the walk's + # sampling keeps one of each. Printed for the record; the assertion is a + # ratio, which a busy laptop slows on both sides alike. + exact = BoostTargets.build() + table = positive_table(3) + sigma = exact.sigma(table) + lines = [] + for point, name in ((4, "round 2"), (2, "draw"), (0, "round 1")): + rows = np.concatenate([row_on(table, point, position) for position in (3, 7, 11)]) + exact.targets(sigma, seat=0, rows=rows[:1], sampling=EXACT) # warm + cost = {} + for label, sampling in (("exact", EXACT), ("walk's sampling", Sampling())): + started = time.perf_counter() + exact.targets(sigma, seat=0, rows=rows, sampling=sampling, seed=0, t=0) + cost[label] = (time.perf_counter() - started) / rows.size + lines.append(f"{name:>8}: exact {cost['exact'] * 1e3:7.1f} ms, walk's sampling {cost["walk's sampling"] * 1e3:6.1f} ms a row") + if point != 4: + assert cost["walk's sampling"] < cost["exact"] / 2, (name, cost) + print("\nboost target, a positive table:\n" + "\n".join(lines)) From b53f2a3ea1685d272cddc9012da3a7f735fd3dce Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Fri, 6 Nov 2026 04:48:39 -0500 Subject: [PATCH 13/16] =?UTF-8?q?fix(rung4):=20every=20chance=20source's?= =?UTF-8?q?=20n/k=20scale=20is=20pinned=20with=20no=20noise=20=E2=80=94=20?= =?UTF-8?q?one=20branch=20at=20a=20time,=20averaged,=20is=20the=20exact=20?= =?UTF-8?q?sum=20to=201e-10=20=E2=80=94=20and=20the=20range=20draw=20is=20?= =?UTF-8?q?tested=20where=20a=20key's=20two=20entries=20face=20different?= =?UTF-8?q?=20ranges;=20the=20stream's=20tail=20is=20two=20words,=20not=20?= =?UTF-8?q?one,=20and=20seed=20and=20t=20take=20any=20integer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/drawmaha_solver/minidrawmaha/AGENTS.md | 6 +- .../minidrawmaha/solver/boost_target.py | 37 ++++--- tests/minidrawmaha/test_boost_target.py | 98 +++++++++++++++++-- 3 files changed, 120 insertions(+), 21 deletions(-) diff --git a/src/drawmaha_solver/minidrawmaha/AGENTS.md b/src/drawmaha_solver/minidrawmaha/AGENTS.md index 0250b052..97aa5019 100644 --- a/src/drawmaha_solver/minidrawmaha/AGENTS.md +++ b/src/drawmaha_solver/minidrawmaha/AGENTS.md @@ -15,9 +15,9 @@ Three subpackages, and one module that stays alone on purpose: frozen σ; the reference), `parallel` (the same in W processes over shared memory), `lockstep_run` (resume, grid checkpoints, progress), and `boost_target` (rung 4's boost target: extra samples of a high-noise row's - regret, each source below the row — board card, both seats' replacements, - the opponent's actions and range — sampled k of n or enumerated as a - `Sampling` names, unbiased, with `EXACT`, the sum over every opponent hand + regret, each source below the row — the board card and both seats' + replacements k of n, the opponent's actions and range k draws — or + enumerated, as a `Sampling` names, unbiased, with `EXACT`, the sum over every opponent hand and every physical entry of the key, as their limit; and the exact probability a walk arrives there; rules-only, and it must never import `exploitability.py`, which grades what it feeds). diff --git a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py index b5066ae6..6188a13f 100644 --- a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py +++ b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py @@ -1,12 +1,12 @@ -"""The boost target: extra, lower-noise samples of a high-noise row's regret, unbiased, with the exact sum as their limit. +"""The boost target: extra samples of a high-noise row's regret, unbiased, with the exact sum as their limit. Rung 4 learns every infoset from the regret samples the walks bank there, `r = u − ⟨σ, u⟩` with one opponent hand and one line of opponent play drawn per visit (`mccfr.traverse`, `lockstep.walk`). At a rare round-2 row a visit comes once in thousands of iterations and its sample swings by chips, almost all of it the opponent's hidden hand. This module draws more samples of that -same quantity at a chosen row, each with less noise than a walk's: it starts -at the row, faces the opponent's whole range (or k hands of it), and plays +same quantity at a chosen row: it starts at the row, faces the opponent's +whole range (or k hands of it), and plays the hand out with each source of randomness below — the board card, both seats' replacements, the opponent's actions — sampled k of n or enumerated, as a `Sampling` names. With every source enumerated (`EXACT`) the target is @@ -14,7 +14,13 @@ tests pin the sampled forms to. A later stage (the "boost") banks the target at the weight the walk's samples would have carried there, so a targeted infoset is learned from a few good samples instead of a handful of noisy -ones. +ones. Against the whole range the opponent's hidden hand — nearly all of a +walk sample's spread — is gone, but a low-k form is not lower-noise than a +walk sample by construction: a k-of-n fan here is public, so it can keep a +branch that is dead for the hand at hand (the board card among a throw's 12 +replacements, a held card among the 14 second-board cards) and scales the +live ones up to pay for it, where the walk deals only live cards. Unbiased +either way; how much noise a form leaves is measured, not assumed. **What is sampled, and why each stays unbiased.** Every sampled source replaces a sum over n branches by k of them, so that the expectation over @@ -37,8 +43,8 @@ reach, back-solved upward, never sampled), and an exact denominator under an unbiased numerator keeps the ratio unbiased; a sampled one would not, since the mean of a ratio is not the ratio of the means. Each row draws from -`default_rng([seed, t, seat, row, "BOOST"])`, its own stream, independent of -the walk's, so a row's target depends on (seed, t, seat, row) and never on +`default_rng([seed, t, seat, row, _STREAM_TAG])`, its own stream, independent +of the walk's, so a row's target depends on (seed, t, seat, row) and never on which rows share the call. **What is computed.** For a row standing for the key I of seat `seat`: @@ -127,7 +133,8 @@ 0.12 s. Exactly, a round-2 row costs about 2 ms, a draw row 160–230 ms and a round-1 row 0.5–0.75 s, because the value below the last two expands over the traverser's own replacements (36 states per throw) and the second board -card (14 boards). Under the walk's own sampling (`Sampling()`) a draw row +card (14 boards). Under the walk's own sampling against the whole range +(`Sampling()`) a draw row costs about 25–32 ms and a round-1 row about 100–130 ms: the board card and the hero's replacements are what sampling cuts. A round-2 row costs more sampled than exact, about 3–4 ms: against a whole range the opponent's @@ -147,6 +154,7 @@ from __future__ import annotations +import numbers from collections.abc import Iterable, Sequence from dataclasses import dataclass from functools import cache @@ -181,8 +189,10 @@ # The widest ledger: the draw's four throws. Regret rows are padded to it. TARGET_WIDTH = max(len(DRAW_ACTIONS), len(BETTING_ACTIONS)) -# The fifth word of a boost stream, ASCII "BOOST": a row's stream can never -# equal a walk's four-word stream or the reservoir's "RESV" one. +# The tail of a boost stream's seed, ASCII "BOOST": 39 bits, which +# SeedSequence takes as two 32-bit words, so a row's stream has six words +# and can never equal a walk's four-word stream or the reservoir's five-word +# "RESV" one. _STREAM_TAG = int.from_bytes(b"BOOST", "big") # --------------------------------------------------------------------------- @@ -789,8 +799,10 @@ def targets( The regret is exact under `EXACT` and an unbiased estimate of it otherwise; the arrival is exact under any `sampling`. A sampled call names `seed` and `t`, and row r draws from - `default_rng([seed, t, seat, r, "BOOST"])` alone, so its target does - not depend on which rows share the call or in what order. A row the + `default_rng([seed, t, seat, r, _STREAM_TAG])` alone, so its target + does not depend on which rows share the call or in what order — and + a row asked for twice gets the same sample twice: more samples of one + row are `playouts`, not repeats. A row the walk cannot reach (zero arrival) gets zero regret, not NaN; a row of the other seat raises, as does one outside the table. """ @@ -798,8 +810,9 @@ def targets( raise TypeError(f"sampling must be a Sampling (EXACT for the exact sum), got {sampling!r}") if not sampling.is_exact: for name, value in (("seed", seed), ("t", t)): - if isinstance(value, bool) or not isinstance(value, int) or value < 0: + if isinstance(value, bool) or not isinstance(value, numbers.Integral) or value < 0: raise ValueError(f"a sampled target needs a non-negative int {name}, got {value!r}") + seed, t = int(seed), int(t) rows = self._rows(seat, rows) d = self.description regret = np.zeros((rows.size, TARGET_WIDTH)) diff --git a/tests/minidrawmaha/test_boost_target.py b/tests/minidrawmaha/test_boost_target.py index 23e0e72e..655d2f14 100644 --- a/tests/minidrawmaha/test_boost_target.py +++ b/tests/minidrawmaha/test_boost_target.py @@ -17,9 +17,13 @@ b and c); rows of the wrong seat are refused and an unreachable row is zero; and the cost per round-2 row is recorded (test d, a benchmark). Then the sampled forms (e): counts that are not positive ints are refused by name; a -count at or above its fan is enumeration, bit for bit; each source alone, -and all of them together, is unbiased against the exact sum at a row where -it is live, and actually moves the target there; a row's sample depends on +count at or above its fan is enumeration, bit for bit; each chance source +kept one branch at a time and averaged over its branches is the exact sum +to 1e-10, in the full game and a restricted one, which pins every n/k scale +with no noise; each source alone, and all of them together, is unbiased +against the exact sum at a row where it is live, and actually moves the +target there; the range draw is unbiased where a key's two physical entries +face wide and different ranges, before and after a throw; a row's sample depends on (seed, t, seat, row) only; play-outs shrink the spread as 1/M; an unreachable row is zero under any sampling; and sampling a draw or round-1 row costs a fraction of its exact sum. @@ -712,9 +716,17 @@ def test_a_count_at_or_above_its_fan_is_enumeration_bit_for_bit(point): # Each source alone at a row where it is live, then all of them at once. The # draw-row cases walk the whole draw and round 2 below them, about 0.2 s a -# call when nothing prunes, so they take fewer seeds. +# call when nothing prunes, so they take fewer seeds. At these seeds the +# smallest relative bias the 4.5-s.e. gate resolves (4.5 · s.e. / |mean|, +# measured 2026-11-06 at these rows) is about 3% for the opponent's actions, +# 8% for the range and 2–8% for the opponent's replacements; for the hero's +# replacements, the board card and the mixed forms it is 13% to over 100%, +# because those sources move the target by more than its size. Their scale +# factors are pinned exactly instead, by the branch oracle below; these +# cases check that the forms compose and that each one actually samples. UNBIASED = [ ("opponent_range", only(opponent_range=1), 4, 0, 400), + ("opponent_range after a throw", only(opponent_range=1), 29, 0, 400), ("opponent_actions", only(opponent_actions=1), 4, 0, 400), ("board_cards", only(board_cards=1), 2, 0, 200), ("hero_draws, seat 0", only(hero_draws=1), 2, 0, 60), @@ -736,16 +748,90 @@ def test_each_sampled_source_is_unbiased_and_actually_samples(sampling, point, s assert arrival[0] > 0.0 draws = sampled(exact, sigma, seat, rows, sampling, range(n)) spread = draws.std(axis=0, ddof=1) - live = spread > 0.0 + live = spread > 1e-9 # above float rounding, which a z would blow up # Not vacuous: the source really varied the target at this row. assert live.any(), "the sampled source never moved the target here" z = (draws.mean(axis=0)[live] - want[0][live]) / (spread[live] / np.sqrt(n)) assert (np.abs(z) < 4.5).all(), z # A column the sample never moves is the exact one, bit for bit or near. - np.testing.assert_allclose(draws[:, ~live], np.broadcast_to(want[0][~live], draws[:, ~live].shape), atol=1e-12) + np.testing.assert_allclose(draws[:, ~live], np.broadcast_to(want[0][~live], draws[:, ~live].shape), atol=1e-9) _, sampled_arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=sampling, seed=0, t=0) assert sampled_arrival[0] == arrival[0] == exact.arrivals(sigma, seat=seat, rows=rows)[0] +def small_draw_row(table: PackedTable, seat: int) -> np.ndarray: + """A draw row reachable from SMALL: seat 0's (both throws below it, two physical entries) or seat 1's.""" + if seat == 1: + return rows_at(table, small_game_keys(1)[:1]) + point = public_decision_points()[2] + return rows_at(table, [key_on(point, canonical_key((SMALL[0][0], (), parse_cards("4c"))))]) + +# Each chance source crosses a path below a draw row once, and the target is +# linear in it, so keeping branch j at every position, scaled n, and +# averaging over j = 0 … n − 1 is the exact sum with no noise at all: a +# scale factor off by one branch in twelve fails at 1e-10 instead of hiding +# inside a standard error. +ORACLE = [ + ("board_cards", 14, None, 0), ("hero_draws", 12, None, 0), ("opponent_draws", 12, None, 0), + ("hero_draws", 12, None, 1), + ("board_cards", 14, SMALL, 0), ("hero_draws", 12, SMALL, 0), ("opponent_draws", 12, SMALL, 0), + ("board_cards", 14, SMALL, 1), +] + +@pytest.mark.parametrize(("source", "fan", "holes", "seat"), ORACLE, + ids=[f"{c[0]}, {'small' if c[2] else 'full'}, seat {c[3]}" for c in ORACLE]) +def test_a_chance_source_kept_one_branch_at_a_time_averages_to_the_exact_sum(source, fan, holes, seat, monkeypatch): + from drawmaha_solver.minidrawmaha.solver import boost_target + exact = BoostTargets.build(holes=holes) + table = positive_table(3) + sigma = exact.sigma(table) + if holes is None: + rows = row_on(table, 2 if seat == 0 else 3) + else: + rows = small_draw_row(table, seat) + want, arrival = exact.targets(sigma, seat=seat, rows=rows, sampling=EXACT) + assert arrival[0] > 0.0 + branch = [0] + monkeypatch.setattr( + boost_target._Draws, "subset", + lambda self, k, n, shape: None if k is None or k >= n else np.full((*shape, 1), branch[0]), + ) + total = np.zeros_like(want) + for j in range(fan): + branch[0] = j + total += exact.targets(sigma, seat=seat, rows=rows, sampling=only(**{source: 1}), seed=0, t=0)[0] + assert np.abs(want).max() > 0.0 + np.testing.assert_allclose(total / fan, want, rtol=1e-10, atol=1e-13) + +# SMALL's seat-0 hole and its clubs ↔ diamonds relabelling against every +# hole holding more clubs than diamonds: a set the relabelling maps onto its +# complement, so the key's two physical entries face wide and different +# ranges (SMALL's three opponent holes leave one live hand at these rows, +# and a range draw there has nothing to vary). +LOPSIDED = ( + SMALL[0], + [hole for hole in HOLES if sum(c.suit == 0 for c in hole) > sum(c.suit == 1 for c in hole)], # suit 0 = c, 1 = d +) + +def test_the_range_draw_is_unbiased_where_two_entries_face_different_ranges(): + # At a stood-pat row and at one after a throw (where the thrown card + # enters the clash mask): a draw that mixed one entry's total or mask + # into the other's would shift the mean. + exact = BoostTargets.build(holes=LOPSIDED) + table = positive_table(4) + sigma = exact.sigma(table) + rows = rows_at(table, small_game_keys(0)) + want, arrival = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) + assert (arrival > 0.0).all() + n = 600 + draws = np.stack([ + exact.targets(sigma, seat=0, rows=rows, sampling=only(opponent_range=1), seed=0, t=s)[0] for s in range(n) + ]) # (n, rows, 4) + spread = draws.std(axis=0, ddof=1) + live = spread > 1e-9 + assert live[0].any() and live[1].any(), spread + z = (draws.mean(axis=0)[live] - want[live]) / (spread[live] / np.sqrt(n)) + assert (np.abs(z) < 4.5).all(), z + def test_a_rows_sample_depends_on_seed_t_seat_and_row_only(): exact = BoostTargets.build() table = positive_table(3) From b4ee6e40d3422a26921fcdc7ae0c179d870ad1f0 Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Fri, 6 Nov 2026 05:01:29 -0500 Subject: [PATCH 14/16] =?UTF-8?q?fix(rung4):=20a=20Sampling=20that=20draws?= =?UTF-8?q?=20nothing=20is=20exact=20whatever=20spells=20it=20=E2=80=94=20?= =?UTF-8?q?None,=20or=20a=20chance=20count=20at=20its=20fan=20=E2=80=94=20?= =?UTF-8?q?so=20it=20needs=20no=20seed=20and=20refuses=20extra=20play-outs?= =?UTF-8?q?;=20counts=20take=20any=20integer=20type,=20as=20seed=20and=20t?= =?UTF-8?q?=20do;=20the=20stream=20comment=20says=20when=20six=20words=20h?= =?UTF-8?q?old?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../minidrawmaha/solver/boost_target.py | 29 +++++++++++++++---- tests/minidrawmaha/test_boost_target.py | 13 +++++++++ 2 files changed, 37 insertions(+), 5 deletions(-) diff --git a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py index 6188a13f..bacb1fe2 100644 --- a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py +++ b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py @@ -190,9 +190,10 @@ TARGET_WIDTH = max(len(DRAW_ACTIONS), len(BETTING_ACTIONS)) # The tail of a boost stream's seed, ASCII "BOOST": 39 bits, which -# SeedSequence takes as two 32-bit words, so a row's stream has six words -# and can never equal a walk's four-word stream or the reservoir's five-word -# "RESV" one. +# SeedSequence takes as two 32-bit words. For seed, t, seat and row below +# 2³² a row's stream is six words, so it cannot equal a walk's four-word +# stream or the reservoir's five-word "RESV" one, and its last two words +# tell it from either at any size. _STREAM_TAG = int.from_bytes(b"BOOST", "big") # --------------------------------------------------------------------------- @@ -201,6 +202,12 @@ _SOURCES = ("opponent_range", "board_cards", "opponent_draws", "hero_draws", "opponent_actions") +# The fans a chance count is read against: the second board card can be any +# of the 14 cards the first left, and a throw's replacement any of the 12 +# cards not in the thrower's hole (dead ones pay zero below). +_BOARD_FAN = len(DECK) - 1 +_REPLACEMENT_FAN = len(DECK) - HOLE_CARDS + @dataclass(frozen=True, slots=True) class Sampling: """How many of each random thing below a boosted row one play-out takes; `None` takes all of them. @@ -235,7 +242,7 @@ def __post_init__(self) -> None: value = getattr(self, name) if value is None and name != "playouts": continue - if isinstance(value, bool) or not isinstance(value, int) or value < 1: + if isinstance(value, bool) or not isinstance(value, numbers.Integral) or value < 1: allowed = "a positive int" if name == "playouts" else "a positive int, or None for all" raise ValueError(f"Sampling.{name} must be {allowed}, got {value!r}") if self.is_exact and self.playouts != 1: @@ -245,7 +252,19 @@ def __post_init__(self) -> None: @property def is_exact(self) -> bool: - return all(getattr(self, name) is None for name in _SOURCES) + """Whether nothing is drawn: every count `None`, or a chance count at or above its fan.""" + return ( + self.opponent_range is None + and self.opponent_actions is None + and all( + k is None or k >= fan + for k, fan in ( + (self.board_cards, _BOARD_FAN), + (self.opponent_draws, _REPLACEMENT_FAN), + (self.hero_draws, _REPLACEMENT_FAN), + ) + ) + ) EXACT = Sampling( playouts=1, opponent_range=None, board_cards=None, opponent_draws=None, hero_draws=None, opponent_actions=None diff --git a/tests/minidrawmaha/test_boost_target.py b/tests/minidrawmaha/test_boost_target.py index 655d2f14..9bfabe26 100644 --- a/tests/minidrawmaha/test_boost_target.py +++ b/tests/minidrawmaha/test_boost_target.py @@ -687,6 +687,9 @@ def test_sampling_refuses_counts_that_are_not_positive_ints(bad): with pytest.raises(ValueError, match=f"Sampling.{field}"): Sampling(**bad) +def test_sampling_counts_take_any_integer_type(): + assert Sampling(board_cards=np.int64(2), playouts=np.int32(3)).board_cards == 2 + def test_exact_with_more_than_one_playout_is_refused(): with pytest.raises(ValueError, match="playouts must be 1"): only(playouts=2) @@ -710,6 +713,16 @@ def test_a_count_at_or_above_its_fan_is_enumeration_bit_for_bit(point): sigma = exact.sigma(table) rows = row_on(table, point) full = only(board_cards=14, opponent_draws=12, hero_draws=99) + # The fans `is_exact` reads are the description's own. + assert exact.description.levels[2].children_of.shape[1] == 14 + assert all( + draw.targets.shape[-1] == 12 + for node_edges in [n.edges for n in exact.description.decision_of_point if n.edges] + for edge in node_edges for draw in edge.draws if draw is not None + ) + assert full.is_exact + with pytest.raises(ValueError, match="playouts must be 1"): + only(board_cards=14, playouts=2) want = exact.targets(sigma, seat=0, rows=rows, sampling=EXACT) got = exact.targets(sigma, seat=0, rows=rows, sampling=full, seed=0, t=0) assert np.array_equal(want[0], got[0]) and np.array_equal(want[1], got[1]) From e5661203cd163155a3cdaf2f53647582e312629a Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Fri, 6 Nov 2026 05:03:03 -0500 Subject: [PATCH 15/16] fix(rung4): the play-outs refusal names both spellings of a Sampling that draws nothing --- src/drawmaha_solver/minidrawmaha/solver/boost_target.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py index bacb1fe2..3625f2c7 100644 --- a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py +++ b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py @@ -247,7 +247,7 @@ def __post_init__(self) -> None: raise ValueError(f"Sampling.{name} must be {allowed}, got {value!r}") if self.is_exact and self.playouts != 1: raise ValueError( - f"Sampling enumerates every source, so every play-out is the same; playouts must be 1, got {self.playouts}" + f"Sampling draws nothing (every count None, or a chance count at its fan), so every play-out is the same; playouts must be 1, got {self.playouts}" ) @property From 47a771831140d67fa48f0c1c448fb6a3890f664b Mon Sep 17 00:00:00 2001 From: Alec Ondo Date: Fri, 6 Nov 2026 05:20:47 -0500 Subject: [PATCH 16/16] fix(rung4): the play-outs refusal says at or above its fan --- src/drawmaha_solver/minidrawmaha/solver/boost_target.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py index 3625f2c7..8830f431 100644 --- a/src/drawmaha_solver/minidrawmaha/solver/boost_target.py +++ b/src/drawmaha_solver/minidrawmaha/solver/boost_target.py @@ -247,7 +247,7 @@ def __post_init__(self) -> None: raise ValueError(f"Sampling.{name} must be {allowed}, got {value!r}") if self.is_exact and self.playouts != 1: raise ValueError( - f"Sampling draws nothing (every count None, or a chance count at its fan), so every play-out is the same; playouts must be 1, got {self.playouts}" + f"Sampling draws nothing (every count None, or a chance count at or above its fan), so every play-out is the same; playouts must be 1, got {self.playouts}" ) @property