Skip to content

fix(ml-worker): cooldown gate for drift-triggered retrains so a storm cannot re-fire every poll cycle (#3168) - #3174

Merged
Xore merged 1 commit into
mainfrom
agent/issue-3168-coder
Sep 12, 2026
Merged

Xore merged 1 commit into
mainfrom
agent/issue-3168-coder

Conversation

@Xore

@Xore Xore commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Drift-triggered retrain used a count-only gate with no time floor. The
window cleared on fire and could refire on the very next poll, which
accepted 50+ retrains in a single hour during the 2026-09-08 storm.

Fix: persisted drift_retrain_allowed() cooldown gate, survives worker
restarts (Europe/Berlin timestamps). Delays retrains only, never mutes
drift alerts. Window still clears only on the accepted-retrain path.

7 new tests added, 305 total pass.

Closes #3168

… cannot re-fire every poll cycle (#3168)

Count-only drift trigger had no time floor: window cleared on fire, then
refired next poll, accepting 50+ retrains in one hour on 2026-09-08.
Adds a persisted drift_retrain_allowed() cooldown gate that survives
worker restarts. Gate only delays retrains, never mutes drift alerts.
Window still clears only on the accepted retrain path.

Closes #3168
@strix-security

strix-security Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Updated for faba351.


Reviewed by Strix
Re-run review · Configure security review settings

@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@strix-security strix-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the PR's two changed files (ml-worker/worker.py and ml-worker/tests/test_model_lifecycle.py, both inlined in the manifest) along with the surrounding run_worker and state-persistence code. The change adds a persisted cooldown gate (drift_retrain_allowed / load_last_drift_retrain / save_last_drift_retrain) plus the DRIFT_RETRAIN_COOLDOWN_MINUTES configuration to throttle drift-triggered retrains, and threads a new drift_due flag through the retrain decision. No security-relevant surface is touched: Elasticsearch access remains parameterized, no secrets or credentials are introduced, and the only new persisted input is internally written UTC state that does not reach any dangerous sink. No injection, authentication, access-control, path, SSRF, serialization, or cryptographic concerns were found in the changed code, so no validation subagents were warranted.


Reviewed by Strix
Configure security review settings

@Xore
Xore merged commit 90b92f2 into main Sep 12, 2026
110 checks passed
@Xore
Xore deleted the agent/issue-3168-coder branch September 12, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ml: drift-triggered retrain storm 2026-09-08 — 50+ retrain-accepted events in under an hour (stability, not threshold)

1 participant