Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/superpowers/plans/2026-11-09-friction-mini-experiment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Friction (RTCFR+-style) on sampled mini-drawmaha: a mini experiment

**Goal.** Does a friction term toward a moving reference make the *current* strategy of
sampled tabular MCCFR converge on mini-drawmaha, and which inner rule carries it best:
CFR+ (W = 10), mini-batch CFR+ (W = 100), or LCFR (W = 10)?

**Done when.** Nine lockstep runs (3 rules × μ ∈ {0, 0.1, 1}) reach 1M hands per seat
in parallel, each graded at 50k / 100k / 250k / 500k / 1M hands on its current strategy
and its linear average, and the results sit in one JSON per run plus a curves figure.
μ = 0 is each rule's no-friction control.

## The friction (Meng et al. 2023, arXiv 2308.11256, Eq. 7–8, adapted to sampling)

RTCFR+ adds μ·(x̂ − r̂) to each player's sequence-form loss: x̂(I,a) is the player's own
reach of sequence (I,a), r̂ the reference's. Its counterfactual value at (I,a) is the
payoff value minus μ·(x̂(I,a) − r̂(I,a)), plus the children's (already transformed) values.

Under external sampling the payoff part of a banked regret is, in expectation, weighted
by the opponent-and-chance reach of I. We apply the friction undivided on each visit
(choice B), so its expected weight carries the same reach: friction and payoff keep the
same ratio at every infoset. The paper's unweighted form would need a 1/reach factor and
explode at rare rows. γ (the paper's 1e-10 mix toward uniform) is numerically inert at
this scale and is left out. Every R hands per seat the reference is reset to the current
strategy; regret totals are kept (the paper: resetting them stops convergence).

## Pieces

1. `solver/friction.py`: `Friction(mu, table, reference)`; `Friction.new(table, mu)`
(reference = current strategy, uniform at the start); `of(key)`; `reset()`.
Vectorised current strategy over a packed table.
2. `mccfr.traverse`: optional `friction`, carrying the traverser's own reach and the
reference's reach down its own actions; utilities at its spots minus μ(x − r).
`friction=None` is today's walk, bit for bit.
3. `lockstep`: `LockstepSolve.friction` handed to every walk; `save_lockstep` and
`ParallelLockstep` refuse a run with friction (not checkpointed, not parallelised).
4. `scripts/minidrawmaha/friction_experiment.py`: one run; resets, grades, JSON lines.
5. Tests (`tests/minidrawmaha/test_friction.py`), written first:
- friction with μ = 0 banks today's regrets bit for bit (Leduc, lockstep W = 4);
- a reference equal to the current strategy along with equal reaches cancels: at t = 1
(all σ uniform, reference uniform) any μ banks today's regrets bit for bit;
- the hand-computed adjustment at one spot;
- the vectorised reset equals `RegretMatcher.strategy` row by row (packed slice);
- save and parallel refuse friction by name;
- Leduc gate (slow): friction + CFR+ drives the current strategy's exploitability
below no-friction CFR+'s current strategy.

## As run (9 Nov 2026)

The machine could not hold nine 1M-hand runs at once (12 concurrent runs land on the
efficiency cores at about 35 hands per seat per second), so the grid ran in waves:
μ ∈ {0, 10} for all three rules to 1M; μ = 0.1 stopped at 50k (indistinguishable
from μ = 0); μ ∈ {1, 100} for CFR+ and LCFR at W = 10 to 500k. The record is
`figures/rung3/friction_mini.jsonl`. Outcome: μ = 10 ends 2–4% below its control at
1M in all three pairs and is 5–11% worse at 50k; μ = 100 is 20–56% worse; the
current strategy beats the linear average at every mark of every arm; LCFR beats
CFR+ by 30%; W = 100 tracks W = 10 within 1.6%.
8 changes: 8 additions & 0 deletions figures/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,11 @@ by `run_deep_cfr.py --rule dcfr` and `--rule cfr+`, its marks graded by
`grades.json`. `rung4/rule_sign_flips.json` is the measurement of why they
lost, written by `scripts/deepcfr/rule_sign_flips.py` from the same runs'
advantage memories.

`rung3/friction_mini.jsonl` is the friction mini experiment's graded record:
every JSON line `scripts/minidrawmaha/friction_experiment.py` appended, one per
grading mark per arm (rule, hands per update W, friction strength μ, current
and linear-average exploitability), concatenated from the arms' output files.
Arms: CFR+ W = 10, CFR+ W = 100 and LCFR W = 10 at μ = 0 and 10 to 1M hands
per seat; at μ = 0.1 to 50k only; CFR+ and LCFR W = 10 at μ = 1 and 100 to
500k. One seed (0) per arm.
49 changes: 49 additions & 0 deletions figures/rung3/friction_mini.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{"rule": "cfr+", "workers": 10, "mu": 0.1, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.5562435935672143, "average": 3.731157099798681, "seconds": 594.4}
{"rule": "cfr+", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.5517268959295873, "average": 3.7356591304923175, "seconds": 576.0}
{"rule": "cfr+", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 3.2509838120613503, "average": 3.417605395721135, "seconds": 1130.8}
{"rule": "cfr+", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 2.5997235419862417, "average": 2.9387484921647653, "seconds": 2253.6}
{"rule": "cfr+", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 1.8008132370632826, "average": 2.333622478692331, "seconds": 3948.0}
{"rule": "cfr+", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 100000, "hands_per_seat": 1000000, "current": 1.0736841309465013, "average": 1.5643887958807077, "seconds": 6909.9}
{"rule": "cfr+", "workers": 100, "mu": 0.1, "reset_hands": 10000, "seed": 0, "iteration": 500, "hands_per_seat": 50000, "current": 3.547235342566566, "average": 3.7194481621097455, "seconds": 618.8}
{"rule": "cfr+", "workers": 100, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 500, "hands_per_seat": 50000, "current": 3.531311741835638, "average": 3.6944582700377397, "seconds": 605.8}
{"rule": "cfr+", "workers": 100, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 1000, "hands_per_seat": 100000, "current": 3.2620827552899994, "average": 3.4310462865215943, "seconds": 1224.0}
{"rule": "cfr+", "workers": 100, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 2500, "hands_per_seat": 250000, "current": 2.61170158222152, "average": 2.9414771241850843, "seconds": 2487.5}
{"rule": "cfr+", "workers": 100, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 500000, "current": 1.777823102998135, "average": 2.3347398527773153, "seconds": 4421.1}
{"rule": "cfr+", "workers": 100, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 1000000, "current": 1.0911306210839766, "average": 1.5658667023559403, "seconds": 7729.8}
{"rule": "lcfr", "workers": 10, "mu": 0.1, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.1816373922487835, "average": 3.374124363882292, "seconds": 605.3}
{"rule": "lcfr", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.1867334340455207, "average": 3.4041814167616087, "seconds": 555.0}
{"rule": "lcfr", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 2.894129293353006, "average": 3.048347266505258, "seconds": 1098.3}
{"rule": "lcfr", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 2.22591541247425, "average": 2.5380436308105256, "seconds": 2189.5}
{"rule": "lcfr", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 1.386015778439032, "average": 1.8762884428553426, "seconds": 3803.6}
{"rule": "lcfr", "workers": 10, "mu": 0.0, "reset_hands": 10000, "seed": 0, "iteration": 100000, "hands_per_seat": 1000000, "current": 0.767706163376485, "average": 1.1307747045097751, "seconds": 6629.3}
{"rule": "cfr+", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.7207998389882166, "average": 3.9205683871690127, "seconds": 591.9}
{"rule": "cfr+", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 3.3316192002878555, "average": 3.5577799505608265, "seconds": 1126.9}
{"rule": "cfr+", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 2.6303120109575158, "average": 3.002654376742573, "seconds": 2309.0}
{"rule": "cfr+", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 1.7795442998298157, "average": 2.348595071065766, "seconds": 4086.8}
{"rule": "cfr+", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 100000, "hands_per_seat": 1000000, "current": 1.043179960227394, "average": 1.5518714019797173, "seconds": 7044.5}
{"rule": "cfr+", "workers": 100, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 500, "hands_per_seat": 50000, "current": 3.705894622009329, "average": 3.9170308452958436, "seconds": 610.3}
{"rule": "cfr+", "workers": 100, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 1000, "hands_per_seat": 100000, "current": 3.3195559232993808, "average": 3.5614458260723794, "seconds": 1173.3}
{"rule": "cfr+", "workers": 100, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 2500, "hands_per_seat": 250000, "current": 2.6109102211910113, "average": 2.994395712916459, "seconds": 2471.8}
{"rule": "cfr+", "workers": 100, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 500000, "current": 1.7855622367234023, "average": 2.350620118255279, "seconds": 4466.7}
{"rule": "cfr+", "workers": 100, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 1000000, "current": 1.0507719008332268, "average": 1.5545607674948498, "seconds": 7895.4}
{"rule": "lcfr", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.552082275982865, "average": 3.7637730030516785, "seconds": 591.3}
{"rule": "lcfr", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 3.1253773110250855, "average": 3.3739519637026376, "seconds": 1132.6}
{"rule": "lcfr", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 2.361427008742529, "average": 2.7362553915378487, "seconds": 2298.9}
{"rule": "lcfr", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 1.5121865175932423, "average": 2.0468756266790447, "seconds": 4053.2}
{"rule": "lcfr", "workers": 10, "mu": 10.0, "reset_hands": 10000, "seed": 0, "iteration": 100000, "hands_per_seat": 1000000, "current": 0.7527588977522754, "average": 1.2125338427647108, "seconds": 6965.3}
{"rule": "cfr+", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.5231938498109763, "average": 3.708012881728954, "seconds": 603.9}
{"rule": "cfr+", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 3.264553079307725, "average": 3.4141293108046815, "seconds": 1090.4}
{"rule": "cfr+", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 2.6076737447371947, "average": 2.9508023108123815, "seconds": 2101.5}
{"rule": "cfr+", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 1.7892240350990598, "average": 2.328847033417266, "seconds": 3620.6}
{"rule": "cfr+", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 4.256367092517985, "average": 4.347168468874717, "seconds": 590.1}
{"rule": "cfr+", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 3.9722703282887686, "average": 4.135957583218573, "seconds": 1057.9}
{"rule": "cfr+", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 3.1379024016058743, "average": 3.5781596647132923, "seconds": 2072.6}
{"rule": "cfr+", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 2.2050062895255187, "average": 2.842050354505566, "seconds": 3594.5}
{"rule": "lcfr", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 3.2001286739426402, "average": 3.417872681156649, "seconds": 608.2}
{"rule": "lcfr", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 2.8856417925374713, "average": 3.0570814178629706, "seconds": 1079.7}
{"rule": "lcfr", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 2.2387650009599414, "average": 2.5537640224399833, "seconds": 2065.7}
{"rule": "lcfr", "workers": 10, "mu": 1.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 1.389492853859736, "average": 1.8915239480846644, "seconds": 3523.9}
{"rule": "lcfr", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 5000, "hands_per_seat": 50000, "current": 4.191457154890394, "average": 4.261162260314845, "seconds": 594.5}
{"rule": "lcfr", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 10000, "hands_per_seat": 100000, "current": 3.9202451147143664, "average": 4.055580788894654, "seconds": 1076.7}
{"rule": "lcfr", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 25000, "hands_per_seat": 250000, "current": 3.1501192015758406, "average": 3.552543871858124, "seconds": 2082.3}
{"rule": "lcfr", "workers": 10, "mu": 100.0, "reset_hands": 10000, "seed": 0, "iteration": 50000, "hands_per_seat": 500000, "current": 2.157896653195825, "average": 2.803019908303629, "seconds": 3609.6}
Loading
Loading