Use flatness to improve SSL for LMs #1

Open

opened

on Aug 16, 2023

Hi, thanks for the amazing work! I'm wondering if you've ever tried to leverage this flatness (Trace of Hessian) as a regularization to improve LM pretraining?

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests