Repository navigation
docs(hunts): state what ships for scaling, smoothing and sigma - #1177
Merged
Merged
Conversation
hunt-runner-scaling.md covers the shim design alone in 1,577 words. It says a starting pod waits for the engine. hunt-schedule-smoothing.md starts with the release status: held back from 2.2.2. It describes an opt-in beta in 2.2.3 behind hunts.smoothing.enabled (#864). docs/data-plane/images/hunt_smoothing_figures.py draws the two ASCII graphs as SVGs. The maths is LaTeX with pseudocode under each block. sigma.md starts with the release status: off behind sigma.enabled until it is tested from feed to detection (#887). It marks each unfinished part where it is described.
The maths section's callout says the maths is hand-written, scanned from a notebook and set in type.
catinspace-au
marked this pull request as draft
November 15, 2026 23:15
Smoothing is off by default. Turn it on with hunts.smoothing.enabled. Turning it off restores the hash offsets. It is not proven at PB/day. The lease is held for one whole cycle, so the fleet runs one cycle per window. A move can come from any bucket above the mean. Hysteresis uses the curve's variance. The blurred tie-break moves a hunt away from similar hunts. A move changes nothing a running hunt holds. Document three odd-window cases in place of two. Add cost_quantile and horizon_max_seconds to the settings table. State which settings are fixed in code. Add the two gauges from the archive to the metrics. Document three tests: - Property tests on the maths. - Tests against a real ClickHouse. - The 30-hunt soak on a 3-replica ClickHouse. Date the soak's flag-off and flag-on numbers. State what the soak does not prove. Refs #864.
The How we know it works section documents the whole run. Across three hours with the flag on, summed peak memory per hour was 6.5, 5.0 and 3.3 GB against 8.1 GB with it off. The busiest bucket held 9, 7 and 8 queries against 12. The model's flatness fell from 23.7 to 2.3 over 34 moves. Refs #864.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three data-plane pages overstate what shipped. Each opens with what's available in 2.2.2.
_waitfor.tpl). 2,303 words down to 1,577.hunts.smoothing.enabled(Adaptive hunt schedule smoothing: drift rate-hunt phases to flatten ClickHouse load #864). What ships is the hashed offset and thelog_commenttagging.docs/data-plane/images/hunt_smoothing_figures.pyreplace the ASCII graphs. They are transparent with mid-grey ink for both themes. The maths is$$LaTeX with pseudocode under each block. The docs site shows the maths as source until dfe-infra adds a maths renderer.sigma.enabled, held back until it's tested from feed to detection (XDR: ship sigma end to end in 2.2.2 (import, author, map, propagate, test) #887). The source mapping works. Providers, the catalogue and propagation are API-only. There are no stock ECS or CIM maps.The three pages must match the 2.2.2 code. The figures must render on the site.