Skip to content

refactor(ema)!: track a module's trainable parameters by reference - #1

Merged
Buddha7771 merged 2 commits into
mainfrom
refactor/ema-module
Oct 1, 2026
Merged

Buddha7771 merged 2 commits into
mainfrom
refactor/ema-module

Conversation

@Buddha7771

@Buddha7771 Buddha7771 commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Make EMA track a module's trainable parameters directly, so callers no longer pass tensors on every call.

Motivation

  • Every caller had to build the tracked set itself ({n: p for n, p in model.named_parameters() if p.requires_grad})
  • update and swap took model.state_dict() on each call, which rebuilds the whole state dict every step only to look tensors up by name
  • Names had to match by string, so a mapping from a wrapped module failed only at run time with a key error

Changes

  • EMA(module, decay) keeps references to the module's requires_grad parameters and raises if there are none
  • update(decay=None) and swap() take no tensors; swap still exchanges in place and swaps back on exit
  • state_dict, load_state_dict, and to are unchanged, and keys stay the parameter names, so saved EMA states still load
  • Bump the version to 0.2.0 for the breaking change

Migration

# before
ema = EMA({n: p for n, p in model.named_parameters() if p.requires_grad})
ema.update(model.state_dict())
with ema.swap(model.state_dict()):
    ...

# after
ema = EMA(model)
ema.update()
with ema.swap():
    ...

Pass the unwrapped module (not a DDP or torch.compile wrapper) so that names match the module's state_dict.

EMA now takes the module and keeps references to its trainable parameters, so
update and swap no longer take a tensor mapping.
@Buddha7771
Buddha7771 merged commit a8ed809 into main Oct 1, 2026
2 checks passed
@Buddha7771
Buddha7771 deleted the refactor/ema-module branch October 1, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant