Skip to content

fix(ml-worker): quarantine malformed events in the retrain path so one bad row cannot kill the worker (#2984) - #3167

Merged
Xore merged 1 commit into
mainfrom
agent/issue-2984-coder
Sep 11, 2026
Merged

Xore merged 1 commit into
mainfrom
agent/issue-2984-coder

Conversation

@Xore

@Xore Xore commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Why

Retrain window data is attacker-influenced. #2978 fixed one crash trigger in that
path; this makes the next unknown one survivable instead of fatal, using the
same caller-side guard pattern as #171.

quarantine_retrain_events() probes each row through both
extract_features() and compute_batch_session_features([row]) before the
retrain proceeds, since either entry point can raise on a malformed event.
Dropped rows are counted and surfaced via write_retrain_metric (with an ES
mapping addition) so quarantine is observable, not silent. Non-retrain readers
stay strict — this isolation is scoped to the retrain path only.

Evidence

5 new isolation tests plus a regression covering the honeypot.session list
shape; full ml-worker suite: 298 passed.

Closes #2984

…e bad row cannot kill the worker (#2984)

Training window data is attacker-influenced. #2978 fixed one crash
trigger there; this covers the next unknown one. Caller-side guard per
#171: quarantine_retrain_events() probes each row through both
extract_features() and compute_batch_session_features([row]) before
the retrain proceeds, since either path can raise on a malformed
event. Dropped rows are counted and surfaced via write_retrain_metric
(plus its ES mapping) so quarantine is visible, not silent. Readers
stay strict elsewhere.

Closes #2984
@strix-security

strix-security Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Updated for ceadfe7.


Reviewed by Strix
Re-run review · Configure security review settings

@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@strix-security strix-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff of ml-worker/worker.py and its accompanying tests. The PR adds per-row crash isolation for the retrain path via quarantine_retrain_events(), which probes each event's _source through the pre-existing extract_features() and compute_batch_session_features() entry points before retraining, drops rows that raise, and records the dropped count in the retrain metric. This reuses the established quarantine pattern from the live scoring path and introduces no new attacker-reachable sinks, no relaxed authorization, and no handling of untrusted data beyond what the unchanged feature extractors already processed inside retrain(). The new dropped_count ES mapping field is a scoped integer addition. Static analysis surfaced three MD5 usages at lines 294, 326, and 785, all outside the changed hunks and therefore out of scope for this review. No security issues were identified in the changed code.


Reviewed by Strix
Configure security review settings

@Xore
Xore merged commit e466289 into main Sep 11, 2026
110 checks passed
@Xore
Xore deleted the agent/issue-2984-coder branch September 11, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ml: ml-worker's retrain path has no per-event isolation — the live scoring path got it in #171, retrain never did

1 participant