fix(ml-worker): replace hard clip in isolation_forest normalization so tail scores stay spread, not pinned at 1.0 (#3169) - #3175
Conversation
…o tail scores stay spread, not pinned at 1.0 (#3169) Clip past p99 collapsed 52.4% of alerts onto identical score 1.0. Swap for continuous slope-matched asymptotic decay: threshold and weights untouched, p50 still maps to 0.2, p99 still maps to 0.95, but tail scores stay spread out past p99 instead of pinning. Also clamps the batch-path exp overflow. The ~20% ceiling acceptance criterion is a post-deploy field measurement, tracked as follow-up. Closes #3169
Strix Security ReviewNo security issues found. Updated for Reviewed by Strix |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
There was a problem hiding this comment.
Reviewed PR #3175, a numerical correction to the isolation-forest score normalization in ml-worker/models/isolation_forest.py (plus its tests). The change replaces the linear extrapolation-with-hard-clip past the p99 anchor with a continuous, slope-matched asymptotic decay toward 1.0, keeping the p50→0.2 and p99→0.95 anchors and the segment slope (0.75) unchanged.
Security review found no vulnerabilities. The modified functions (_percentile_normalize and _percentile_normalize_batch) take only internally-derived model scores and calibration percentiles (p50/p99 from the model's own holdout data), not untrusted web/request input, and return bounded values in [0,1]. The only numerical hazard — exp() overflow in the vectorized np.where path — is explicitly mitigated via np.maximum(frac - 1.0, 0.0) before the exponent, and the scalar path's exponent is provably non-positive on its branch. Division is guarded by the p99 == p50 early return. No auth, authorization, secret, injection, or serialization surfaces are touched. The test file changes add only assertions and the already-imported math module. Static analysis (bandit, gitleaks) reported no findings.
Reviewed by Strix
Configure security review settings
Clip past p99 collapsed 52.4% of alerts onto identical score 1.0.
Replaces the hard clip with continuous slope-matched asymptotic decay:
threshold and weights untouched, p50 still maps to 0.2, p99 still maps
to 0.95, but tail scores stay spread out past p99 instead of pinning.
Also clamps the batch-path exp overflow.
301 tests pass, including new discriminating saturation tests.
The ~20% ceiling acceptance criterion is a post-deploy field
measurement (Europe/Berlin), tracked as follow-up.
Closes #3169