Skip to content

Add synthetic data replay ratio support - #103

Merged
Oaklight merged 1 commit into
mainfrom
feature/synthetic-replay-ratio
Sep 26, 2026
Merged

Oaklight merged 1 commit into
mainfrom
feature/synthetic-replay-ratio

Conversation

@Oaklight

Copy link
Copy Markdown
Owner

Summary

Replaces #102 (auto-closed during branch cleanup).

  • Add synthetic_ratio and synthetic_sources parameters to SamplerConfig
  • Controls proportion of synthetic vs benchmark data per epoch (e.g. 0.4 = 40% synthetic, 60% benchmark)
  • Inspired by CLM paper showing 40% replay prevents catastrophic forgetting
  • Per-type allocation: splits each type's budget between synthetic and benchmark sources
  • Graceful fallback: if a type has no synthetic sources, all items come from benchmark (and vice versa)
  • Auto-detection: sources with "synthetic" in name/path are auto-tagged if synthetic_sources not explicitly set
  • Wired through YAML config and CLI (--synthetic-ratio 0.4)
  • 5 new tests covering ratio control, edge cases, and fallback behavior

Closes part of #97 (synthetic replay ratio approach)

Combined correctness verification

All 3 data mixing features (temperature + difficulty_weights + synthetic_ratio) compose correctly:

=== All three combined ===
  Types: {'score': 1715, 'choice': 857, 'noul': 428}
  Score/noul ratio: 4.0x ✓  (difficulty_weights working)
  Synthetic: 899/3000 (30%) ✓  (synthetic_ratio working)
  Temperature controls within-type source distribution ✓

Test plan

  • python -m pytest tests/test_sampler.py -v — 22 tests pass (all 3 features combined)
  • Combined composition test: temperature + difficulty + synthetic all interact correctly

Add configurable synthetic_ratio to control the proportion of synthetic
vs benchmark data in each epoch (CLM-inspired replay). When set, each
type's allocation is split between synthetic and benchmark source groups
with weighted sampling within each group.

- SamplerConfig: add synthetic_sources and synthetic_ratio fields
- MultitaskSampler: add _split_sample method, use in sample_epoch
- train_multitask.py: wire --synthetic-ratio CLI arg and config parsing
  with auto-detection of synthetic sources by name/path
- All 5 multitask YAML configs: add commented-out synthetic_ratio option
- 5 new tests covering ratio control, edge cases, and fallback behavior
@Oaklight
Oaklight merged commit d7f7a3d into main Sep 26, 2026
4 checks passed
@Oaklight
Oaklight deleted the feature/synthetic-replay-ratio branch September 26, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant