Loss starts to diverge When the discriminator opens

[cbyzju]

I try to train VQGAN on UCF-101 dataset with 4 A100s, and 24 samples for each device, recontruction and perceptual loss could converge normally, until the discriminator is opened in 10k steps. In addtion, the commitment loss diverge from the start time, How can I fix it?

[wdrink]

Same issue here. Expect the authors to provide the script to reproduce the results on UCF-101🙏

[songweige]

What does the reconstruction look like at these steps? I could try to provide more helps after the ECCV deadine!

[wdrink]

It seems the model learns well until the discriminator is opened in 10k steps. After the discriminator is opened, the reconstructed images are blurred w/o any texture and even semantics.

[songweige]

Hi @wangjk666, sorry for the late follow-up! I hope you have resolved the issue but just in case not, I have some suggestions on what might help. So the GAN loss is notoriously hard to tune but it can get better/sharper results in my experience. We also discussed a bit about this in the appendix A.1. Perhaps you could also refer to some more modern techniques people use to stabilize discriminative losses, e.g. ADD Sec 3.2.

[Dongzhikang]

Same issue here. GAN loss even crashes the result.