Coeff Question

[efittschen]

I had a question regarding why we have coeff, and why its implemented the way it is.
In forward we calculate different kl_divergences,

var	#	comment
kl_alpha	topic_num * time_slice_num	(kl divergences summed)
kl_eta	time_slice_num	(kl divergences summed)
kl_theta	batch_size * coeff	(kl divergences summed)
nll	batch_size * coeff	(losses summed)

---->

var	#	comment
kl_theta	batch_size * #dataset_docs/#batch_size	(kl divergences summed)
nll	batch_size * #dataset_docs/#batch_size	(losses summed)

So kl_theta and nll are scaled up depending on training dataset size.

That means that the larger the training dataset is, the less important it becomes that kl_alpha and kl_eta stay within the gaussian walk prior. Which doesnt seem like it is what we want?

I was wondering if this might explain why I felt the topic word distributions seemed to sometimes jump from one topic to another topic in the scifi project.

relevant code:

def forward(self, bows, normalized_bows, times, rnn_inp, num_docs):
    ...
    bsz = normalized_bows.size(0)
    coeff = num_docs / bsz 
    ...
    kl_theta = kl_theta.sum() * coeff
    ...
    nll = nll.sum() * coeff
    nelbo = nll + kl_alpha + kl_eta + kl_theta
    return nelbo, nll, kl_alpha, kl_eta, kl_theta

loss, nll, kl_alpha, kl_eta, kl_theta = model(
                                                  ...
                                                  len(train_docs),
                                              )

[efittschen]

If I understand it correctly the motivation behind coeff might be that we dont want the scaling of kl_theta and kl_nll to be dependent on batch size, but making it dependent on training dataset size seems wrong as well.

[TomLippincott]

I think, for theta, the idea is to first get the average per-document divergence according to the current batch (by dividing by bsz), and then treating that loss as though it's being observed for the full corpus, thus weighting it higher than the eta/alpha KLDs. Same goes for the reconstruction loss. I guess the effect might be to amplify regularization/reliance on the priors for the less-observed mixtures compared to the more general ones (eta, alpha), which are allowed to more easily diverge.