Bug: worker marks completed session failed retroactively when group-overall step errors

[Alimedhat000]

Summary

In ml/tasks.py run_inference, a session is committed as ml_status="completed" with its score before the group-overall step runs. If the group step then fails, the except handler retroactively flips the already-completed session to failed and nulls its score.

Evidence

• ml/tasks.py:143-146 — session score/status committed as completed first
• ml/tasks.py:151-198 — group computation + predict_overall() (can raise, e.g. ValueError on NaN/Inf)
• ml/tasks.py:246-256 — except handler sets session.ml_score = None, ml_status = "failed", and commits

Impact

A group-level failure (not a session-level one) permanently fails a valid inference:

• RQ retry of the job hits the ml_status == "failed" guard at ml/tasks.py:110 and aborts (RuntimeError), so retries never re-run.
• Recovery is manual via the reset endpoint.

Suggested fix

Separate session-level and group-level failure handling. Suggested approach: keep the session's committed score/status on group-step failure, record the group as failed (or retry only the group step), and only mark the session failed when its own inference actually failed. Preserve the existing RQ retry semantics.

Verification

Unit-test the worker with a session whose own inference succeeds but whose group completion throws (mock predict_overall to raise), and assert the session keeps ml_status="completed" and ml_score while the group is failed.