`update_ensemble!` hangs on all-NaN G_ensemble

[costachris]

When all columns of g are NaN, update_ensemble! hangs indefinitely. Ideally this can throw an error right away. I believe this results in DataMisfitController.calculate_timestep! producing NaN timesteps, leading to sqrt of an all-NaN covariance matrix which never returns.

minimal reproducer:

using EnsembleKalmanProcesses as EKP, LinearAlgebra

obs_vec = [EKP.Observation(Dict(
    "samples" => randn(60), "covariances" => Diagonal(0.001*ones(60)), "names" => "obs_$i",
)) for i in 1:6]
obs_series = EKP.ObservationSeries(obs_vec, EKP.RandomFixedSizeMinibatcher(6, "trim"), ["obs_$i" for i in 1:6])
ekp = EKP.EnsembleKalmanProcess(randn(10, 100), obs_series, EKP.Inversion();
    scheduler=EKP.DataMisfitController(on_terminate="continue"),
    failure_handler_method=EKP.SampleSuccGauss())
EKP.update_ensemble!(ekp, fill(NaN, 360, 100))  # hangs

[odunbar]

Possible solution

We do preprocessing of each iteration to impute NaNs, so this should be an easy fix to add if_all_nan throw(ArgumentError()) statement

EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl

Lines 1312 to 1315 in 0e87d7d

	function impute_over_nans(ekp::EnsembleKalmanProcess, g::AM) where {AM <: AbstractMatrix}
	nan_row_values = isnothing(get_nan_row_values(ekp)) ? get_obs(ekp) : get_nan_row_values(ekp)
	return impute_over_nans(g, get_nan_tolerance(ekp), nan_row_values, verbose = ekp.verbose)
	end

Update: See PR - I found a nicer place to do the check