In order to allow to run more experiments of the same model safely in parallel, we need to implement lockfiles. These files would have all the necessary fields in order for check_params to work and block running more experiments with the same parameters concurrently. One consideration to keep in mind, the start of the jobs in the array should be somewhat randomized at the beginning as there is potential for collision, however I have yet to found a simple argument for this in the SLURM docummention.
In order to allow to run more experiments of the same model safely in parallel, we need to implement lockfiles. These files would have all the necessary fields in order for
check_paramsto work and block running more experiments with the same parameters concurrently. One consideration to keep in mind, the start of the jobs in the array should be somewhat randomized at the beginning as there is potential for collision, however I have yet to found a simple argument for this in the SLURM docummention.