Repository navigation
feat(rung4): the boost stage — one sampled play-out per targeted infoset after the walk's rows, at (1 − α)·K·arrival with the walk's rows turned to α, so expected banked weight and regret are plain sampling's; 35 s an iteration at B = 500, never exact - #94
AlecROndo wants to merge 59 commits into
Conversation
…every opponent hand and every physical entry of its key, and the walk's arrival rate, from the rules alone PR 1 of the targeted-sampling plan. `minidrawmaha/solver/exact_target.py` computes, for any table row, the regret the walk's samples there estimate (the mean over the opponent's hands of u − ⟨σ, u⟩, weighted by the deck and the opponent's own play, normalised by the arrival) and the probability one walk visits the row. It re-derives the public tree, key maps, draw maps and showdown from rules/ and never imports the grader, which grades what this will feed. Measured: a round-2 row costs 0.8 ms, the arrival alone 0.3 ms; a draw row 45–85 ms and a round-1 row 0.2–0.6 s, because the value below expands over the traverser's own replacements and the second board card. Tests: the back-solved reach equals the grader's carried vector on every board of every point (1e-10, full and restricted deals); arrival at a seat's first decision is the grader's frequency and sums to one; regret and arrival equal a world-by-world walk through MiniState on a small restricted game (1e-9) and the mean and visit count of 6,000 Collector walks row by row within 4 s.e. and pooled; the module's import block holds no grader.
…, 5,190 vectors, re-derived by a test; the sampled tests select rows by expected count, so a row cannot enter because it fluctuated high Review follow-ups on #89. The docstring counted the memo's level-1 points alone (29) where the chance nodes on the paths are memoised too (66 in all); the test now walks the ancestor closure and pins 66, 20 and 5,190. `reach()` hands out a copy. The pooled z² windows are a 4-sd band on the chi-square mean, √(2/n), instead of fixed numbers. The gated 200,000-walk test kept rows by observed count ≥ 100, which let seat 1's draw rows in only when they had fluctuated high (mean z +1.8 on two seeds); rows are now kept by expected count ≥ 80 against an observed cutoff of 40, and both seats read mean z within 0.01 of zero over thousands of rows. The module docstring names the double-visit caveat and no longer points at a plan outside the repo without saying where it is.
… the mean of z², and cap a single row at six standard errors — the deal alone dealt one picture 292 times against 220 at seed 12 The pooled mean is the sharp bias test (a tenth of a standard error in every row is caught at 1,600 rows); the per-row cap is loose because of about 3,000 rows one passes 5 in a run out of a thousand, and the deal itself did it at seed 12 in a count that involves nothing of this module. Run with MINIDRAWMAHA_FULL_TARGET=1: both seats pass.
…e band's assumptions written beside it — the "caught at 1,600 rows" line was a half-truth; the band is reached there and reliable at about 3,000
…9 at n = 10, biting from n ≈ 45 — and the arrival seeds checked named
…width drops under 1, not from 45
…caller says otherwise; a file without the column loads as ones PR 2 of the targeted-sampling plan, first of three commits. The reservoir's row grows from (row, t, target) to (row, t, target, weight): 24 to 28 bytes, a 50M-row memory 1.2 to 1.4 GB. The weight rides its row's slot through Algorithm R's eviction, save and load; merge without a weight stamps ones; a file with no weight key is from before the column and loads with ones. Refused by name: a negative, non-finite, mis-shaped weight on merge, and a weight array of the wrong length or dtype on load. The learner seam still takes three arrays, so tests/deepcfr/test_train.py is red until the next commit, which threads the column through it.
…y's loss bit for bit and a row of weight zero is inert Second of PR 2's three commits. _weighted_loss scales each row's squared error by (t · w) / mean(t · w): with w ≡ 1 that is t / mean(t) in the same bits, since t · 1.0 is t in float32. fit_from_scratch and batch_loss take the weight as a fourth positional array, None meaning ones, and refuse a weight that is not 1-D, not the memory's length, non-finite, negative, or all zero (nothing to fit, as an empty memory is). A batch whose every row weighs 0 divides by a clamped mean and has loss 0, not NaN. fit_policy takes the column positionally so the policy memory's contents() unpack straight in; its weights are all one.
…cle banks Σ t′ · w · r in LCFR's order, so weight one keeps the pin bit for bit Last of PR 2's three commits. Learner.fit takes the memory's four arrays, the weight positional with no default so no learner can drop it by accident; step is unchanged, the four-tuple from contents() flowing through. NetLearner hands the weight to fit_from_scratch and the held batch's loss, after the pot scaling and the regret rules act on the target: the weight says how loudly a row counts, never what it says. The oracle multiplies each sample by its weight in float64 before the per-t′ np.add.at sum, where × 1.0 is exact, so the LCFR pin passes unchanged; a weight of 2 banks what the row offered twice would, a weight of 0 nothing. Oracle model files carry the weight and load as ones without it.
… refused by name — the ones default ran ahead of the checks it relied on; and the weight-one pin is against the old loss formula, not the code itself From the fresh-context review before the push. Reservoir.load sized the default weight from a negative `offered` before refusing it; merge and fit_from_scratch sized it from a 0-d rows before the 1-D check; a 0-d weight on load indexed its own empty shape. Each now fails on the by-name ValueError the old code gave. test_weight_one_is_the_unweighted_fit_bit_for_bit compared the fit to itself (both paths built the same ones); it now restates the pre-column loss, t / mean(t) over the legal squared error, and pins batch_loss at weight one and by default against it to the bit, then the whole fit.
…ight for overflow, on merge, on load and in the fit — a float64 weight past float32's range passed the finite check and was stored as inf; and the oracle's "offered twice" claim holds to the bit only when the two are alone at the column - `_validate_offer` casts to float32 first and checks finite, non-negative and t × weight finite after; a bool or non-numeric weight is refused by name. - `Reservoir.load` refuses a weight that is NaN, negative or overflows t × weight, naming the file, as `merge` never writes one. - `fit_from_scratch` refuses a t × weight that overflows float32, the loss's scale. - `OracleLearner` docstring: weight 2 equals the row offered twice exactly only when nothing else sits at the column in that t′; otherwise up to rounding, since `np.add.at` sums one at a time. A new test pins two rows of widths 2 and 3 with weights 3 and 2 against the rows offered three and two times, so a weight landing on a neighbour's columns would show. - `batch_loss` says why it defaults the weight while `Learner.fit` does not; `NetLearner.fit` notes the held loss is a log line nothing decides on.
…high-noise row — board card, both seats' replacements, the opponent's actions and range — each unbiased against the exact sum, which stays as EXACT, with the arrival always exact and one stream per (seed, t, seat, row) exact_target.py becomes boost_target.py (ExactTargets → BoostTargets). targets() takes a required Sampling; EXACT reproduces the previous output bit for bit.
… one branch at a time, averaged, is the exact sum to 1e-10 — and the range draw is tested where a key's two entries face different ranges; the stream's tail is two words, not one, and seed and t take any integer
…le's rows — 141 lines and 3,870 line × hand classes, from the rules alone
… √(E[r²]/p)/|E[r]|, from the memory's own walk rows only — an exact row is a mean, not a sample
…nfoset and take them until the pool is 20 × B reached rows; infoset takes the B best outright
…rrival by exponential keys, reproducible from (seed, t, seat, stream) and never the same row twice
…rozen target set, asking for arrivals at the pool alone; the Refresher protocol is the seam PR 4 calls
…hecked on every import statement
… — None, or a chance count at its fan — so it needs no seed and refuses extra play-outs; counts take any integer type, as seed and t do; the stream comment says when six words hold
…ound 1's and hold 19% of LCFR-2M's exploitability on 0.07% of its reached infosets; the 20 leakiest hold 57% on a quarter
…nto AlecROndo/pr-3-targeting
…that draws nothing
…ds as zero; a real negative is still refused — the LCFR-2M release has gains at −7e-12
…, by choice, the current strategy), weighs the pool by the walk's arrival, and is the only importer of the meter on the trainer's side
…nto AlecROndo/pr-3-targeting
|
Claude finished @AlecROndo's task in 49s —— View job Review of PR #94 (boost stage)I read 1. "Removes noise, never adds emphasis" is not guaranteed (medium severity, medium confidence).
2. "The reservoir sees the same offers as a plain run" is overbroad (low severity, high confidence).
3. The weight guard is not randomised (low severity, medium confidence;
4. Deferred items (agree with the PR's disposition).
5. Tests (low severity, medium confidence).
6. Pedagogy and scope nits.
What I checked and found fine.
Not run: |
…lay-out's variance is under a walk sample's over K·p, a boosted run matches a plain one in expectation not row for row, α = 1 is refused, step refuses a boost without an aim, and the sampling field is boost_sampling
|
Round 1 of the review, dispositioned in c3a0135.
Tests: |
|
Claude finished @AlecROndo's task in 22s —— View job Review of PR #94 (boost stage), updated for the current headI read The earlier review on this PR was written against an older commit. These are now resolved in the head:
What I re-derived and found correct
Remaining findings
Pedagogy. The weight derivation and the all-or-nothing treatment argument are correct and clear. Not run: |
|
Round 2, dispositioned in 6d23ead (docstring only; an earlier version of this comment named c3a0135 before the commit had landed).
|
…pacity, that a zero-arrival row's play-out is never run, and wraps its lines
|
Claude finished @AlecROndo's task in 29s —— View job Review of PR #94 (boost stage), at head 6d23eadI read Resolved since the earlier rounds. I checked each of these against the current
Re-derived and correct.
Findings
Not verified. I did not check whether Not run: |
…ogress line reports them as rows.turned_down, in place of a capacity figure the docstring had wrong
|
Round 3, dispositioned in 3ec9080.
|
|
Claude finished @AlecROndo's task in 22s —— View job Review of PR #94 (boost stage), at head 3ec9080I read the diffs of 6d23ead and 3ec9080 in Resolved since my last round
Remaining findings, all low severity
Not verified: that Not run: |
|
Claude finished @AlecROndo's task in 26s —— View job Review of PR #94 (boost stage), at head 1d03e30I read the diff of the two newest commits (3ec9080, 1d03e30) in Round 3 is resolved correctly.
Remaining findings, all low severity.
Not run: |
…ned_down's that α > 0 down-weights rather than silences
|
Round 4, dispositioned in bd08963 (docstring only).
The touched test files were re-run after every commit ( |
|
Claude finished @AlecROndo's task in 16s —— View job Review of PR #94 (boost stage), at head bd08963I read the diff of bd08963, which touches only Resolved from my last round
Still open, all low severity and all for PR 5, as the PR body already says
Since the last two commits changed only docstrings, the 1,921-pass result still applies. I ran no tests myself. |
…fers, with rows.boosted beside it
|
Round 5, dispositioned in c10d5e9 (one docstring line).
|
|
Claude finished @AlecROndo's task in 32s —— View job Review of PR #94 (boost stage), at head c10d5e9I read the diff of c10d5e9 (a One docstring problem in the new text (low severity, medium-high confidence; The new sentence says The next clause is muddled: "the walk's own rows of the iteration are
Suggested wording: " If that is wrong because Open items for PR 5, unchanged from earlier rounds and already in the PR body
No other new findings. |
… the iteration's boost counts
|
Round 6, dispositioned in 1a45946 (one docstring paragraph). Agreed: |
|
Claude finished @AlecROndo's task in 14s —— View job Review of PR #94 (boost stage), at head 1a45946I read the diff of 1a45946, which changes one docstring in Resolved from my last round. The
One caveat, low severity and medium confidence. The difference rule needs consecutive log lines. The same paragraph says a resumed run re-appends iterations, and a reader should take the last line per t. Differencing must happen after that dedup, and the first iteration has no predecessor. The wording is correct as written. You could add "after taking the last line per t" if you want it airtight. Still open for PR 5, all already in the PR body
No other new findings. |
|
Round 7: agreed and no change — the paragraph already tells a reader to take the last line per t, and the difference rule applies after that; the first iteration's predecessor is zero offers. Nothing else open on this PR; the three PR 5 checks are in the body and with the PR 5 agent. |
PR 4 of the targeted-sampling plan for rung 4: the boost stage, wired into
train.step. After a seat's walks and before the merge, the stage draws this iteration's B target rows from PR 3's frozenTargetSet, computes one sampled boosted row per target under the σ the walks just read (PR 1'sBoostTargets.targetswith aSampling), turns the walk's own rows at those infosets down to α, and offers the boosted rows after the natural ones at weight (1 − α) · K · arrival (PR 2's column). A targeted infoset's expected banked weight and regret are plain sampling's; whether its noise falls is what the race will measure (it does where one play-out's variance is under a walk sample's over K · p). A run without an aim is today's run bit for bit.Branched from #93's head (PR 3), with
mainmerged in twice — once at the start and once after #93 landed as a squash, with this branch's side taken where PR 4 had edited PR 3's lines — never rebased. The diff againstmainis byte-identical to PR 4's diff over PR 3's head.Decisions, settled by Alec on 2026-11-09
TargetSpec's defaults.TargetSpecfield,leak, so a resume refuses a change to it by name.Sampling()by default: the whole opponent range in one vectorised sum, one second board card, one replacement per throw, one opponent action line.DeepConfigrefuses an exactSamplingandinjectrefuses it again;EXACTis the tests' oracle only. The opponent-range count is a flag, so the mini-walk form (one dealt hand per play-out, M play-outs) is one argument away. Alec's rule: boosting is extra sampling at high-variance nodes, never brute-forcing the tree from a node.What it adds
src/drawmaha_solver/deepcfr/boost.py(new, NumPy only, never imports the grader)inject(rows, targets, *, drawn, exact, sigma, seat, t, seed, sampling, alpha, walks) -> Injected: the phase's advantage offer with one boosted row per treatable drawn row appended. Natural rows first, in walk order, at weight α where their row is treated and 1 elsewhere; boosted rows after, at(1 − α) · K · arrivalin float32. A drawn row is treated all or nothing: one with zero arrival, or whose float32 weight would equal exactly 1 or α (PR 3's noise score tells walk rows from boosted rows by weight alone), is left untreated and counted inskipped. Refuses an exactSampling, a repeated or unsorteddrawn, a bad α or K.Boost(refresher, exact=None):due(targets, t),refresh(t=, table=, memories=, spec=)(refuses a set built for another t or another spec),phase(targets, offered, table=, seat=, t=, seed=, sampling=, alpha=, walks=), which reads σ from the table handed to it and drawsrows_for(t, seat, seed). The whole game'sBoostTargetsis built on first use (~5 s).train.pyDeepConfig.target: TargetSpec | None = NoneandDeepConfig.boost_sampling: Sampling = Sampling(); both inADDED_LATER(None, the default sampling) andSAMPLE_FIELDS; both travel as nested dicts instate.jsonand are rebuilt byconfig_from_saved, which refuses a nested dict missing a field or carrying an unknown one by name.step(..., boost=None): materialise → observe → (seat 0, when due) refresh → collect → inject → merge (with the weight) → fit. Without a boost the merge call is unchanged.DeepRun.targets;save_run/load_runcarrytargets-t{t:05d}.npzwhen the run has an aim; an old state without it loads withNone.run_deep(..., boost=None)refuses an aim without a boost and a boost without an aim, before the collector opens.StepStatsand the progress line gainseconds.refresh,seconds.boost[2],rows.boosted[2],rows.skipped[2],rows.turned_down[2](zeros without an aim).targeting.py:TargetSpec.leak("average"|"current"),budget=500,refresh=25; theRefresherprotocol is__call__(*, t, table, memories)— whatever else a refresher needs it is given when built, so the loop hands over only what it has.scripts/deepcfr/targeting_refresh.py:GraderRefresher(spec, *, walks, average=None, seed=0, exact=None)readsspec.leak;averageis required when the aim follows the average's leak and unused otherwise.scripts/deepcfr/run_deep_cfr.py:--boost {none,random,infoset,line,line_class}and--boost-budget / -refresh / -alpha / -leak / -pool / -noise-cap / -round-two-only, plus the sampling of one boosted row,--boost-playouts / -opponent-range / -board-cards / -opponent-draws / -hero-draws / -opponent-actions(an int orall). An exact sampling is refused at the command line. With an arm named,mainbuildsGraderRefresheraround the run'sExactAverageand hands the loop aBoost. (--targetwas already the iteration to reach, hence--boost.)deepcfr/AGENTS.md(the no-EXACT rule, the resume set, the config fields),deepcfr/__init__.py(eleven modules),scripts/AGENTS.md,train.py's iteration sketch and timings,reservoir.py's andtargeting.py's docstrings (the boosted row is sampled, not exact), one README sentence, and the plandocs/superpowers/plans/2026-11-09-targeted-sampling-pr4-boost-stage.md.Measured
Whole-game smoke (
MINIDRAWMAHA_FULL_DEEP=1, the line arm at B = 500, K = 200, the oracle learner,leak="current"), this Mac:About 35 s per iteration at B = 500, 500 rows a seat boosted and none skipped; at R = 25 the refresh adds about 4 s per iteration. That is half again over the 23 s PR 3's per-row costs predicted for B = 500 (and about one and a half of the plan's 18 to 23 s table iterations); B stays 500, as decided, and the gap is recorded in
train.py's timings. The plan's B = 2,000 would be about 140 s. Seat 0 costs more because its draw lands more often on round-1 rows, whose play-out crosses the second board card and the draws.An aim flag (
--boost-budgetand the like) with--boostleft at none is refused at the command line, a fresh-context review finding: it would otherwise be today's run with nothing in the log.Tests
uv run pytest(whole default suite, no torch): 1,932 passed, 25 skipped at the head after the review rounds (1a45946's code; the last commits are docstrings), 20 min 32 s with another suite run competing for the machine; 1,921 passed in 10 min 32 s at 72c3946 before the review fixes added eleven CLI and guard tests.test_the_oracle_loop_is_lcfr_bit_for_bitpasses unchanged.tests/deepcfr/test_boost.py(24 tests):injectagainst hand-worked numbers: order, the weights at α = 0 and 0.5, a drawn row with no natural rows, the zero-arrival skip, the exactly-1-or-α skip at both α, the refusal of EXACT and of an exact-by-countsSamplingbefore any target is computed, repeated/unsorted/non-integer drawn rows, fewer rows than B.Boost:due,refresh's refusals,phasereading σ from the table it is handed and drawing this iteration's rows, the lazy whole-game oracle.BoostTargets, one iteration at 24 seeds (50 underMINIDRAWMAHA_FULL_TARGET=1) with and without the boost, same walks: at every targeted row (two opening rows and the four busiest later rows) the paired difference in banked regret is within 4 s.e. of zero at every legal column, the banked weight difference is within 4 s.e. of zero, the boosted weight is K · arrival to the bit every seed, and the boosted rows' mean is within 4 s.e. of the EXACT regret. Mutation checks (not committed): a K + 1 weight and natural rows left at weight 1 each fail it. About 30 s.tests/deepcfr/test_train.py(+17): the config fields and their refusals,ADDED_LATER/SAMPLE_FIELDS, the nested JSON round trip, an old state without the fields, the refresh cadence (t = 1, 3, 5 at R = 2), boosted rows after the natural ones with the policy memory untouched (realBoostTargets), σ read after each phase's write (a stamping learner), the progress line, the target set in the resume and marks (not re-graded; refreshed on resume once expired), stop-and-restart equals straight, the tworun_deeprefusals, the CLI flags and the exact-sampling refusal,main's wiring, and the gated whole-game smoke.test_targeting.py,test_targeting_refresh.py:leak, the new defaults, the refresher's constructor-time average.Departures from the appendix
--boost…, not--target…(--targetis the iteration to reach).Refreshertakes noaverage;GraderRefresheris built with it.TargetSpec.leakexists so the leak source is part of the saved run.Not in this PR
PR 5, the race (launcher, grading, verdict, pre-registration), which should also record, per targeted row, one play-out's variance σ_b² against a walk sample's over the visits, σ_w² / (K · p): the boost is unbiased everywhere and quieter only where that inequality holds, and the race must tell "the boost helps" from "the boost is merely unbiased". Its pre-registration should also check that
rows.skippedstays near 0 and readrows.turned_down(the walk's rows made inert at α = 0, the boost's cost in reservoir slots once eviction starts) at the chosen K. Nothing has been run with the boost beyond the smoke.Pre-push review (fresh-context reviewer subagent), dispositioned
No correctness bug found; the weight arithmetic, the float32 guard, the σ timing, the offer order, the streams and the resume state were each re-derived and checked against the tests. Findings:
targeting_refresh.py's docstring still said the boost stage "computes exactly"; it now says one sampled play-out per row.TargetSpec's docstring,train.py's timings and a test comment.train.py's sketch counted two boost lines; it has three.--boostleft at none was silently dropped; the parser now refuses it by name (7 new CLI cases).steprefuses a boost when the run has no aim, by name.injectandTargetSpec.Set aside by the reviewer and ruled on:
modal_rung4.pyhas no--boostpath (PR 5's); the first refresh grades an exact average of one observation (a consequence ofleak="average");rows.advantagein the log now counts boosted rows among the offers, withrows.boostedbeside it.