Runs inconsistent even with static random seed (Integration tests no longer working) #380
Description
Hi @dav-sonn @Mariia-Var @igelstorm - following on from conversations yesterday it does seem that with the merging of branches the integration tests are failing consistently now and never able to produce stable estimates between runs, in some cases producing 5-10% gaps in output statistics under identical runs (same seed, same starting conditions). Have tried running this with bootstrapping on and off and still producing discrepancy. In latest updates there seems to be some new process introducing variation between runs which is not dependent on the random seed set at the start of the run.
There's a couple of issues:
- integration tests can now no longer pass and wont be able to detect problematic changes in outputs amongst between-run expected noise
- no runs are now reproducible under a set random seed
The first problem would a simple fix of removing the integration test checks for consistent values (as currently implemented in develop and main branches). However, the second issue remains in that I'd presume runs are intended to be identical for set random seeds? I'm not sure where this could have been introduced.