fix(ml-worker): quarantine malformed events in the retrain path so one bad row cannot kill the worker (#2984) - #3167
Conversation
…e bad row cannot kill the worker (#2984) Training window data is attacker-influenced. #2978 fixed one crash trigger there; this covers the next unknown one. Caller-side guard per #171: quarantine_retrain_events() probes each row through both extract_features() and compute_batch_session_features([row]) before the retrain proceeds, since either path can raise on a malformed event. Dropped rows are counted and surfaced via write_retrain_metric (plus its ES mapping) so quarantine is visible, not silent. Readers stay strict elsewhere. Closes #2984
Strix Security ReviewNo security issues found. Updated for Reviewed by Strix |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
There was a problem hiding this comment.
Reviewed the full diff of ml-worker/worker.py and its accompanying tests. The PR adds per-row crash isolation for the retrain path via quarantine_retrain_events(), which probes each event's _source through the pre-existing extract_features() and compute_batch_session_features() entry points before retraining, drops rows that raise, and records the dropped count in the retrain metric. This reuses the established quarantine pattern from the live scoring path and introduces no new attacker-reachable sinks, no relaxed authorization, and no handling of untrusted data beyond what the unchanged feature extractors already processed inside retrain(). The new dropped_count ES mapping field is a scoped integer addition. Static analysis surfaced three MD5 usages at lines 294, 326, and 785, all outside the changed hunks and therefore out of scope for this review. No security issues were identified in the changed code.
Reviewed by Strix
Configure security review settings
Why
Retrain window data is attacker-influenced. #2978 fixed one crash trigger in that
path; this makes the next unknown one survivable instead of fatal, using the
same caller-side guard pattern as #171.
quarantine_retrain_events()probes each row through bothextract_features()andcompute_batch_session_features([row])before theretrain proceeds, since either entry point can raise on a malformed event.
Dropped rows are counted and surfaced via
write_retrain_metric(with an ESmapping addition) so quarantine is observable, not silent. Non-retrain readers
stay strict — this isolation is scoped to the retrain path only.
Evidence
5 new isolation tests plus a regression covering the
honeypot.sessionlistshape; full ml-worker suite: 298 passed.
Closes #2984