Hi, thank you for open sourcing your work.
I think in the training script training/svd_runner.py, there's a missing code at line 578:
accelerator.backward(loss)
+ optimizer.step() # add this line, or the weight never gets updated
lr_scheduler.step()
optimizer.zero_grad()