Skip to content

Add difficulty-proportional type ratio override - #101

Merged
Oaklight merged 1 commit into
mainfrom
feature/difficulty-proportional-sampling
Sep 26, 2026
Merged

Oaklight merged 1 commit into
mainfrom
feature/difficulty-proportional-sampling

Conversation

@Oaklight

Copy link
Copy Markdown
Owner

Summary

  • Add difficulty_weights parameter to SamplerConfig as a semantic override for type_ratios
  • Allows weighting type sampling inversely proportional to accuracy (harder types get more data)
  • Example: {noul: 0.5, choice: 1.0, score: 2.0} — score (~50% acc) gets 4x the data of noul (~80% acc)
  • Different from per-type LR (Multi-task optimization: per-type LR, gradient balancing, adaptive loss weighting #95, didn't help) — this controls data quantity, not learning rate
  • Wired through YAML config and CLI (--difficulty-weights '{"noul": 0.5, "choice": 1.0, "score": 2.0}')
  • 3 new tests covering override behavior and edge cases

Closes part of #97 (difficulty-proportional sampling approach)

Test plan

  • python -m pytest tests/test_sampler.py -v — 12 tests pass (9 existing + 3 new)

@clementine-oaklight clementine-oaklight Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ✅ — Clean difficulty-proportional sampling.

Component Status
Override ✅ difficulty_weights overrides type_ratios when set
Fallback ✅ effective_ratios = difficulty_weights or type_ratios
CLI ✅ --difficulty-weights '{...}' JSON string
Tests ✅ 3 tests (override, fallback, missing type)

Addresses data quantity (vs per-type LR which only adjusted gradient scale). CI green.

@milo-oaklight milo-oaklight Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved ✅

Difficulty-proportional type ratio override:

Design

  • difficulty_weights overrides type_ratios for sampling
  • Controls data quantity per type, not learning rate
  • Inverse-accuracy weighting: harder types get more training data

Example

difficulty_weights:
  noul: 0.5     # ~80% accuracy → low weight
  choice: 1.0   # ~61% accuracy → medium weight  
  score: 2.0    # ~50% accuracy → high weight (4x noul)

Implementation

  • effective_ratios = self._config.difficulty_weights or self._config.type_ratios
  • Clean override pattern — no special casing scattered through code
  • Used in both items_per_type allocation and window slot allocation

CLI

  • --difficulty-weights '{"noul": 0.5, "choice": 1.0, "score": 2.0}'
  • Parsed as JSON string

Tests (3 new)

  • Override behavior (score gets ~4x noul)
  • None falls back to type_ratios
  • Missing type handled gracefully

CI green.

@elena-oaklight elena-oaklight Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Clean implementation of difficulty-proportional sampling.

Review Summary

Aspect Status
CI ✅ All 4 checks passed
Logic ✅ Correct override mechanism
Tests ✅ 3 new tests cover key scenarios
Config ✅ YAML + CLI wiring complete

Implementation Notes

  1. Override pattern is sound: effective_ratios = difficulty_weights or type_ratios — simple, explicit, no partial merge.

  2. Consistent application: effective_ratios used in all 4 locations:

    • Initial type allocation (L174, L182)
    • Window slot allocation (L220, L227)
  3. Missing-type handling: Test confirms graceful behavior when difficulty_weights references absent types (score in noul+choice pool).

  4. CLI integration: --difficulty-weights '{"noul": 0.5, "choice": 1.0, "score": 2.0}' parses correctly via json.loads.

  5. Config docs: Commented examples in all 5 YAML configs provide usage guidance without changing defaults.

Minor observation (not blocking): The docstring says "overrides type_ratios for sampling" — this is accurate for proportions but doesn't affect the window-interleaving structure itself, which is expected.

Add difficulty_weights field to SamplerConfig that overrides type_ratios
when set, allowing harder question types (lower accuracy) to receive
more training data. Wire through train_multitask.py with both YAML
config and --difficulty-weights CLI support. Add commented-out examples
to all 5 multitask config files.
@Oaklight
Oaklight force-pushed the feature/difficulty-proportional-sampling branch from a527725 to 5264509 Compare September 26, 2026 22:46
@Oaklight
Oaklight merged commit 5e6c9a7 into main Sep 26, 2026
4 checks passed
@Oaklight
Oaklight deleted the feature/difficulty-proportional-sampling branch September 26, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant