feat(arena): weight model selection toward under-voted models#448
Draft
simonaszilinskas wants to merge 1 commit into
Draft
feat(arena): weight model selection toward under-voted models#448simonaszilinskas wants to merge 1 commit into
simonaszilinskas wants to merge 1 commit into
Conversation
Leaderboard statistics get unreliable when some models have 10k battles and others have ~100. Tilting selection slightly toward under-voted models helps vote counts converge so comparisons rest on similar sample sizes. Selection uses w_i = 1 / (n_match_i + smoothing)^alpha, normalized, with the max/min weight ratio capped at 3x. n_match comes from the RankingResult already cached in Redis by the leaderboard pipeline, so no new DB load. pick_two fetches once and passes the ranking to both pick_one calls. Defaults alpha=0.4, smoothing=50, max_ratio=3.0 are tunable via WEIGHTED_SAMPLING_* settings, with an ENABLED kill-switch. Methodology tab on the ranking page gets a short section explaining the bias in fr/en/da/sv.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leaderboard stats get unreliable when some models have 10k battles and others ~100. This tilts selection slightly toward under-voted models so vote counts converge and comparisons rest on similar sample sizes.
Selection uses
w_i = 1 / (n_match_i + smoothing)^alpha, normalized, with the max/min weight ratio capped at 3x.n_matchcomes from theRankingResultalready cached in Redis by the leaderboard pipeline, so no new DB load.pick_twofetches the ranking once per call and reuses it across bothpick_onepicks.Defaults are
alpha=0.4,smoothing=50,max_ratio=3.0, all tunable viaWEIGHTED_SAMPLING_*settings, with anENABLEDkill-switch in case anything looks off in prod.The Methodology tab on the ranking page has a short new section explaining the bias (fr/en/da/sv).
Draft for now, want to eyeball real staging numbers before marking ready. Plan:
weighted_pickat debug) for a day or twoTests:
backend/llms/tests/test_weighting.pycovers ordering, ratio clip, zero-vote smoothing, flag-off, and a Monte-Carlo distribution check. Run with.venv/bin/python -m unittest backend.llms.tests.test_weighting.