gaussian grbm initialization

[jquetzalcoatl]

grbm weights and biases initialization set to Gaussian N(0,1/number of nodes)

Hinton guide suggests 0.01 as standard deviation. See https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

Moreover, having it set to Gaussian with this dependence on the number of nodes makes the energy extensive and initializes the gRBM in a paramagnetic phase similar to that describen in the Random Energy model paper

https://journals.aps.org/prb/abstract/10.1103/PhysRevB.24.2613

See #48

[kevinchern]

@jquetzalcoatl IIRC, Hinton's recommendation pertains to zero-one-valued RBMs (bipartite with hidden units). Would it make sense to translate the $0.01$ to the spin-valued equivalent?

[jquetzalcoatl]

@kevinchern The REM reference is for spin models i.e., {-1,1}. Ultimately, the initialization pertains to whether the model is ergodic. In this sense, the support only set an offset energy.

I believe the main motivation for initializing with 0.01 in Hinton's guide is to start in a paramagnetic phase, which ties nicely with the REM/SK spin glass model

[kevinchern]

Can you add a release note to go with this?

[jquetzalcoatl]

added release note

[kevinchern]

Following motivation from references, should h be initialized to 0?

[kevinchern]

Tests are failing but otherwise LGTM. Thanks for the much-needed PR @jquetzalcoatl !!

@VolodyaCO offered to take a look at the tests