Skip to content

feat(rung3): friction at μ = 10 ends 2–4% below its control at 1M hands — the current strategy already beats the average without it - #95

Open
AlecROndo wants to merge 3 commits into
mainfrom
alec/rtcfr-friction-mini
Open

AlecROndo wants to merge 3 commits into
mainfrom
alec/rtcfr-friction-mini

Conversation

@AlecROndo

Copy link
Copy Markdown
Owner

What

RTCFR+-style friction (Meng et al. 2023, arXiv 2308.11256) in the sampled tabular walk on mini-Drawmaha. The goal is to see whether a pull toward a moving reference makes the current strategy converge, under three inner rules: CFR+ (W = 10), mini-batch CFR+ (W = 100) and LCFR (W = 10).

  • solver/friction.py: Friction(mu, table, reference) and a vectorised reset. At the traverser's spots the tax μ·(own reach·σ − reference reach·reference) is subtracted from each action's value. It is added undivided on each visit, so it is reach-weighted like the payoffs.
  • mccfr.traverse and lockstep take an optional friction. friction=None and μ = 0 are today's walk bit for bit (tested). save_lockstep and ParallelLockstep refuse a friction run by name.
  • scripts/minidrawmaha/friction_experiment.py: runs one arm, resetting the reference every 10k hands. It grades the current and average strategies at each mark; --save-table writes the arrays for row analysis.
  • figures/rung3/friction_mini.jsonl: every graded mark of every arm (one seed each).

Findings (exploitability, chips per hand)

μ = 0 at 1M μ = 10 at 1M μ = 10 at 50k μ = 100 at 500k
CFR+ 1.074 1.043 (−2.8%) +4.8% +22%
Mini-batch CFR+ 1.091 1.051 (−3.7%) +4.9% —
LCFR 0.768 0.753 (−1.9%) +11% +56%
  • μ = 0.1 and μ = 1 changed nothing (within 0.8%).
  • Without friction, the current strategy beat the linear average at every mark of every arm, by 1.44–1.47× at 1M. The average lags about a third of the run behind, so there is no circling yet for friction to remove.
  • W = 100 tracked W = 10 within 1.6%: a rarely reached row still gets one sample per update, so the CFR+ floor still acts on single noisy samples.
  • Row diagnostics at 1M: never-sampled rows, which play uniform, hold under 1% of the leak, and visited-but-uniform rows hold 8–10%. About 90% sits in learned rows, so friction (which cannot act at unvisited rows) has little to work on.

Tests

uv run pytest: 1882 passed, 25 skipped. The gated Leduc gate (MINIDRAWMAHA_FULL_CALIBRATION=1) passes: on Leduc, friction makes CFR+'s current strategy beat plain CFR+'s.

https://claude.ai/code/session_014D1ZwBFFxJAkajfcQPe6bx

…ampled walk — μ = 0 is today's walk bit for bit, and on Leduc it makes CFR+'s current strategy converge

Claude-Session: https://claude.ai/code/session_014D1ZwBFFxJAkajfcQPe6bx
…ds and costs 5–11% before that — the current strategy already beats the average at every mark without it

The friction experiment script gains --save-table (the finished table's regrets,
strategy sums, widths and reference, for row-level analysis), and the graded
record of every arm lands in figures/rung3/friction_mini.jsonl: CFR+ W = 10,
CFR+ W = 100 and LCFR W = 10 at μ ∈ {0, 10} to 1M hands per seat, μ = 0.1 to
50k, μ ∈ {1, 100} for CFR+ and LCFR W = 10 to 500k. One seed per arm.

Claude-Session: https://claude.ai/code/session_014D1ZwBFFxJAkajfcQPe6bx
@vercel

vercel Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
drawmaha-solver Ready Ready Preview Nov 9, 2026 1:14pm UTC

Request Review

@claude

claude Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Claude finished @AlecROndo's task in 1m 26s —— View job


Review of PR #95 (friction mini experiment)

  • Read the diff (friction, mccfr, lockstep, parallel, script, tests)
  • Recomputed the PR table from figures/rung3/friction_mini.jsonl
  • Posted inline comments

I did not run the test suite. I only read the code and the jsonl.

Numbers check

Every figure I recomputed from the jsonl matches the PR body:

  • μ=10 vs μ=0 at 1M: −2.8% (CFR+ W=10), −3.7% (W=100), −1.9% (LCFR).
  • μ=10 at 50k: +4.8%, +4.9%, +11%.
  • μ=100 at 500k: +22% and +56%.
  • Average/current ratio at 1M: 1.44–1.47×.

Findings

# Where Finding Confidence Severity
1 friction.py L21–28 The claim "tax and payoff keep the paper's ratio" is wrong. The paper does not weight the tax by π₋ᵢ, but here it is weighted, so the tax is weaker at rarely reached rows. The next sentence contradicts the claim. medium-high medium (pedagogy)
2 Same file, L3 and L13 "Current strategy circles forever" is overbroad, and this PR's own data contradicts it. The "zeroing regrets stops convergence" citation is unverified. medium / low low
3 test_friction.py There is no end-to-end test of the reach plumbing in traverse. The only checks are equality at t=1 and != after 20 iterations, so a sign error would pass. The "bit for bit" claim is not pinned to pre-PR bytes. Packed-table and vanilla paths are untested, and so is the negative-μ refusal. The Leduc gate is one seed, a bare <, skipped by default, with an unjustified μ=0.05. medium medium
4 friction_experiment.py L7 The docstring mentions a --hands flag that does not exist. high low
5 Headline claim The "2–4% below control" result comes from one seed per arm, with no noise estimate. It is also mixed: at 500k, CFR+ W=100 is +0.4% at μ=10, LCFR W=10 is +9%, and LCFR's average at 1M is ~7% worse (1.2125 vs 1.1308). The title and body should say the effect is within an unmeasured seed spread, or add a second seed for the μ=0 and μ=10 pair. medium medium
6 mccfr.py friction.of(state.infoset()) repeats the slot lookup that table[state.infoset()] just did. This is a minor cost on every traverser visit. Friction.table is typed object. low low

Overall

  • The sign convention (subtract μ(x̂ − r̂) from utility) agrees with RTCFR+'s loss-side formulation. The reset, the refusals in save_lockstep and ParallelLockstep, and the keyword-only handling of the existing reach weights look right.
  • The null result matches the weakening described in finding 1. A natural follow-up is to test a variant that divides the tax by the row's visit probability, or one that uses a clipped version.
  • The PR is a correct negative or weak result with an honest write-up. The docstring math in finding 1 should be fixed in this change, per AGENTS.md ("a wrong claim in a docstring is a bug").
    · branch alec/rtcfr-friction-mini

Comment on lines +21 to +28
One departure from the paper, forced by sampling. In the paper the friction enters
every infoset's value with full weight, every iteration. External sampling banks a
row only when it is visited, and the payoff part of what it banks is weighted, in
expectation, by the chance-and-opponent reach of the row. The friction here is added
undivided on each visit, so it carries the same reach weight: at every row the tax
and the payoff keep the ratio they have in the paper, but the tax is reach-weighted.
Dividing by the visit probability instead would recover the paper's weighting and
hand rare rows enormous, noisy shoves. The paper's γ (a 1e-10 mix toward uniform)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstring claim looks wrong (confidence: medium-high, severity: medium — pedagogy). "at every row the tax and the payoff keep the ratio they have in the paper" does not follow from what is written.

In the paper, a row's counterfactual regret is weighted by the opponent and chance reach π₋ᵢ, but the friction term μ(x̂ − r̂) is not. Their ratio at a row is therefore (tax)/(π₋ᵢ·payoff).

Here both are added undivided per visit. In expectation both are scaled by the visit probability, so the ratio is the paper's multiplied by π₋ᵢ. The tax is weaker at rarely reached rows than in the paper, by the factor π₋ᵢ.

The next sentence ("dividing by the visit probability would recover the paper's weighting") says the same thing, so the two sentences contradict each other. This is also a plausible mechanism for the null result at rarely reached rows, and the PR's own row diagnostics point that way. Suggest rewording to: "the tax is down-weighted at row I by π₋ᵢ(I) relative to the paper, so μ is an effective μ·π₋ᵢ".

Line 3 (confidence: medium, severity: low): "A CFR run's CURRENT strategy circles the equilibrium forever" is overbroad. This PR's own data shows the current strategy beating the average at every mark. "need not converge, and in the worst case circles" is safer.

Line 13 (confidence: low, severity: low): "the paper reports that zeroing them stops convergence" — I could not verify this against arXiv 2308.11256 from here. Please cite the section or soften the claim.

--out runs/friction-mini/cfrplus-w10-mu0.1.jsonl

Trains one run in this process (`train_lockstep`; friction is not parallelised) to
`--hands` hands per seat. Every `--reset-hands` hands per seat it moves the friction's

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confidence: high, severity: low. The docstring mentions --hands, but no such flag exists. The budget is the last of --marks. Also, the run terminates at marks[-1], so "Trains to ..." should say so.

@@ -0,0 +1,130 @@
"""Friction toward a reference (RTCFR+-style) in the sampled walk.

* **Off is today.** `friction=None` and friction at μ = 0 bank exactly today's regrets.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test gaps (confidence: medium, severity: medium).

  • "Off is today" compares friction=None with μ=0, i.e. two new code paths. It does not compare against the pre-PR walk. That holds by construction, because the None branch is the old code, but "bit for bit" is not pinned to any frozen bytes. It is also only tested on dict tables with lcfr and cfr+, not on a PackedTable or the vanilla rule.
  • No test checks that traverse's reach plumbing (own_reach * sigma[k], reference_reach * reference[k]) matches a hand-computed tax over a small tree. friction_adjustment is tested in isolation. The only end-to-end tests are "equals at t=1" and "!= after 20 iterations". The second asserts only inequality, so a wrong sign or a missing reach would pass. A one-decision toy game with known regrets would pin this down.
  • Friction.new's negative or non-finite μ refusal is untested.
  • test_the_reset... samples keys[::37] and only key 0 falls in the first 40 all-negative columns, so the uniform-fallback branch is barely exercised.
  • The gated Leduc test is one seed, a bare < with no margin, and μ=0.05 with no justification. It is skipped by default, so the default suite never checks that friction does anything useful.

…π₋ᵢ against the paper, and tests pin its sign, μ refusals and the uniform reset branch

Review follow-up on #95: the module docstring said the sampled tax keeps the
paper's tax-to-payoff ratio; it keeps a constant ratio, i.e. μ acts as μ·π₋ᵢ(I).
New tests: strong friction toward a fixed uniform reference leaves Leduc play
nearer uniform than plain play; negative or non-finite μ is refused; the reset
check covers the all-negative (uniform) rows. The script docstring drops a
nonexistent --hands flag.

Claude-Session: https://claude.ai/code/session_014D1ZwBFFxJAkajfcQPe6bx
@AlecROndo

Copy link
Copy Markdown
Owner Author

Addressed in cb0f4aa:

  • friction.py docstring: now says the sampled tax is down-weighted by π₋ᵢ(I) relative to the paper (μ acts as μ·π₋ᵢ), softened the 'circles forever' line, and says regrets are kept across resets as in the paper's algorithm instead of claiming a reported result.
  • Script docstring: dropped the nonexistent --hands; the run stops at the last of --marks.
  • Tests: new end-to-end sign/reach check (strong friction toward a fixed uniform reference leaves Leduc play nearer uniform than plain), negative/NaN/inf μ refused, reset check now covers the all-negative uniform rows, Leduc gate's μ = 0.05 justified.
  • Not done: a frozen-bytes pin of the pre-PR walk (the None branch is the old code unchanged) and a hand-built one-decision toy game; the sign test above covers the wrong-sign/missing-reach failure mode.

@claude

claude Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Claude finished @AlecROndo's task in 35s —— View job


Re-review of PR #95 after the two follow-up commits

I read the code and did not run the test suite.

Earlier findings, status

  • Docstring (π₋ᵢ weighting): fixed. friction.py now says the tax is down-weighted by π₋ᵢ(I) relative to the paper, so μ acts as an effective μ·π₋ᵢ(I). The stated mechanism holds up. Both the payoff and the tax are added per visit, so the tax-to-payoff ratio is the same at every row.
  • "Circles forever" and the unverified "zeroing regrets" citation: both fixed. The first is softened and the second is dropped.
  • --hands in the script docstring: fixed. It now says the run stops at the last of --marks.
  • Test gaps: mostly closed.
    • New tests cover the μ refusal, a hand-computed adjustment, the reset, a pinned sign, and the refusals in save_lockstep and ParallelLockstep.
    • Still open, low severity: "bit for bit" is shown by comparing None against μ=0, not against frozen pre-PR bytes. The None branch is the old code, so this holds by construction.
    • Still open: the Leduc gate is one seed, skipped by default, and uses μ=0.05 without a stated reason.

Remaining nits (all low)

  1. Packed-table path through traverse: Friction.of is only exercised end to end on dict tables. A tiny packed-table equality check at μ=0 would cover the production path. Confidence: medium.
  2. Headline claim: the 2–4% gain at 1M is one seed per arm, with no noise estimate. The title says "2–4% below its control", so a second seed for the μ=0 and μ=10 pair would show whether that is signal. The PR body does state the 50k and 500k costs honestly. Confidence: medium.
  3. Repeated slot lookup: friction.of(state.infoset()) repeats the slot lookup that table[state.infoset()] just did. This is a small cost per traverser visit. Friction.table is typed object. Confidence: low.
  4. Formatting: the docstring line "RTCFR+ (Meng et al. 2023, arXiv 2308.11256, Eq. 7–8) changes" and the γ line run past the wrapped width. There is only one blank line before the top-level defs in friction.py. This is cosmetic.

Verdict: the math and docstrings now agree with the code, and the sign convention matches the paper's loss-side formulation. I have no blocking findings. The one-seed caveat is the main thing I'd still address before merge.
· branch alec/rtcfr-friction-mini

This branch was successfully deployed

1 active deployment
Preview — cb0f4aa0 Deployed Nov 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant