Summary
During the first training iteration (when reinit is true), scHPF allocates a temporary random_phi Dirichlet matrix to initialize Xphi_data. That array is not deleted afterward, so it can linger until the next GC cycle and inflate peak memory use—especially with large batches / many nonzeros.
Proposed fix
Delete random_phi immediately after it is used to construct Xphi_data:
Xphi_data = X_batch.data[:,None] * random_phi
del random_phi # free memory
Summary
During the first training iteration (when
reinitis true), scHPF allocates a temporaryrandom_phiDirichlet matrix to initializeXphi_data. That array is not deleted afterward, so it can linger until the next GC cycle and inflate peak memory use—especially with large batches / many nonzeros.Proposed fix
Delete
random_phiimmediately after it is used to constructXphi_data: