From c890437d5e12d342a49e7deb56ab04fa59abbbbd Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 08:46:56 +1100 Subject: [PATCH 1/4] docs(hunts): state what ships for scaling, smoothing and sigma hunt-runner-scaling.md covers the shim design alone in 1,577 words. It says a starting pod waits for the engine. hunt-schedule-smoothing.md starts with the release status: held back from 2.2.2. It describes an opt-in beta in 2.2.3 behind hunts.smoothing.enabled (#864). docs/data-plane/images/hunt_smoothing_figures.py draws the two ASCII graphs as SVGs. The maths is LaTeX with pseudocode under each block. sigma.md starts with the release status: off behind sigma.enabled until it is tested from feed to detection (#887). It marks each unfinished part where it is described. --- docs/data-plane/hunt-runner-scaling.md | 294 +++------- docs/data-plane/hunt-schedule-smoothing.md | 407 +++++--------- .../images/hunt-smoothing-before-after.svg | 506 ++++++++++++++++++ .../images/hunt-smoothing-cpu-vs-memory.svg | 328 ++++++++++++ .../images/hunt_smoothing_figures.py | 125 +++++ docs/data-plane/sigma.md | 16 +- 6 files changed, 1185 insertions(+), 491 deletions(-) create mode 100644 docs/data-plane/images/hunt-smoothing-before-after.svg create mode 100644 docs/data-plane/images/hunt-smoothing-cpu-vs-memory.svg create mode 100644 docs/data-plane/images/hunt_smoothing_figures.py diff --git a/docs/data-plane/hunt-runner-scaling.md b/docs/data-plane/hunt-runner-scaling.md index a05cb9c9..a5aa23d6 100644 --- a/docs/data-plane/hunt-runner-scaling.md +++ b/docs/data-plane/hunt-runner-scaling.md @@ -6,59 +6,28 @@ Copyright: (c) 2026 HYPERI PTY LIMITED --> -# Hunt-runner scaling: deterministic-due, no engine dependency - -The hunt runner is pull-based and multi-pod. **Default deployment is ONE steady pod** -(no ScaledObject at all) - the runner is a thin orchestrator (all hunt load -runs in ClickHouse), so one pod covers the large majority of deployments. -Autoscaling is **BETA and opt-in** (`huntRunner.autoscaling.enabled`). When -turned on, KEDA scales the pod count on the deterministic-due backlog this doc -explains, via the fail-safe `dfe-keda-shim` - never the engine, and no longer -a MySQL-on-ClickHouse scaler (retired, see "KEDA ScaledObject" below). - -**Keep autoscaling off (the default).** It reads the due backlog from -the `hunt_schedule` table, and nothing in the charts writes that table: -there is no materialise Job, so the backlog always reads 0. With autoscaling on -the runner never scales out, and at `minReplicaCount: 0` no hunt ever fires. -To try it anyway, run `dfe-hunt-runner materialise` after every hunt-config -change. - -This doc is the SCALING contract: how the due backlog is computed with zero -ambiguity, how the beta ScaledObject wakes/grows/shrinks the pod count from -it, and the property that matters most - the runner has NO operational -dependency on dfe-engine. - -## The one hard requirement: due-detection with zero workers - -This is the contract the BETA autoscaling relies on (see "Default deployment, -beta autoscaling, and disable" below for when it applies - it also governs -correctness if you opt into `minReplicaCount: 0`, i.e. scale-to-zero). KEDA -can only wake the first worker if "is any hunt due right now?" is answerable -with no worker running - i.e. as a query KEDA itself (via the shim) runs -against ClickHouse. But the per-hunt schedule (interval + stable phase offset) -otherwise lives only in a running worker's memory (loaded from the gitops -hunt configs). At zero workers there is nobody to ask. - -So the schedule is MATERIALISED into a small ClickHouse table, `hunt_schedule`, at -config-deploy time. The scaler then reads it directly. The runner's other tables -(`src/dfe_engine/hunts/ch_coordinator.py`) are `hunt_lease`, `hunt_watermark`, -`hunt_state`, `hunt_run` and `hunt_runner_heartbeat`. `hunt_schedule` is the only -one written outside the worker loop. +# Hunt-runner scaling + +The hunt runner is pull-based and multi-pod. The default deployment is ONE steady pod with no ScaledObject, because the runner is a thin orchestrator and all hunt load runs in ClickHouse. One pod covers the large majority of deployments. + +Autoscaling is a beta, off by default (`huntRunner.autoscaling.enabled`). Leave it off. It reads the due backlog from the `hunt_schedule` table and nothing in the charts writes that table, so the backlog always reads 0. With autoscaling on, the runner never scales out, and at `minReplicaCount: 0` no hunt ever fires. To try it anyway, run `dfe-hunt-runner materialise` after every hunt-config change. + +When on, KEDA scales the pod count on the deterministic-due backlog described below, read through the fail-safe `dfe-keda-shim`. KEDA never asks the engine. + +## Due-detection with zero workers + +KEDA can only wake the first worker if "is any hunt due right now?" is answerable with no worker running, as a query the shim runs against ClickHouse. The per-hunt schedule (interval and stable phase offset) otherwise lives only in a running worker's memory, loaded from the gitops hunt configs. + +So the schedule is materialised into a small ClickHouse table, `hunt_schedule`, at config-deploy time: ```text hunt_schedule(hunt_id, interval_seconds, phase_offset, enabled, updated) ReplacingMergeTree(updated) ORDER BY hunt_id ``` -`enabled` is a soft tombstone: a deleted hunt is written `enabled=0` so it stops -waking KEDA, without a DELETE. - -## The scaler query (deterministic-due) +`enabled` is a soft tombstone: a deleted hunt is written `enabled=0` so it stops counting as due, without a DELETE. The other coordination tables (`src/dfe_engine/hunts/ch_coordinator.py`) are `hunt_lease`, `hunt_watermark`, `hunt_state`, `hunt_run` and `hunt_runner_heartbeat`. `hunt_schedule` is the only one written outside the worker loop. -KEDA runs `schedule.due_query(db)` against ClickHouse (via the fail-safe -`dfe-keda-shim`'s `/keda/hunt-backlog` endpoint - see "KEDA ScaledObject" -below) - the count of hunts that are due-and-unclaimed. It uses the EXACT -arithmetic the worker uses (`schedule.latest_fire`): +The shim's `/keda/hunt-backlog` endpoint runs `schedule.due_query(db)`, which counts hunts that are due and unclaimed. It uses the same arithmetic the worker uses (`schedule.latest_fire`): ```text boundary = intDiv(now(), interval) * interval -- start of this interval @@ -69,127 +38,49 @@ due = watermark < fire -- that fire not already co AND lease_until <= now() -- not currently running ``` -`due_count > 0` -> scale up, and `== 0` -> scale down (to zero only where -`minReplicaCount: 0` has been opted in - the default floor is 1 pod). The -`enabled=1` filter is applied in an inner subquery so a tombstone -(interval=0) can never reach the `intDiv` (no divide-by-zero). - -### Why the watermark comparison is exact - -The worker writes `watermark = scheduled_start` (the fire time) after a run commits -(`worker.py`: `window(...)` returns `end = scheduled_start`, then -`set_watermark(end)`). So the watermark is in the SAME epoch-fire scale as the -boundary arithmetic - `watermark < fire` means precisely "this fire has not been -completed". No unit mismatch, no data-time-vs-fire-time skew. - -## Why it is RELIABLE - -Autoscaling (when the BETA is enabled) is only safe if KEDA's view of "due" -never diverges from a worker's. It cannot, because both compute the same -thing from the same inputs: - -1. Parity, tested. The scaler predicate equals `runner.tick`'s own gate (via - `schedule.latest_fire` + the watermark + lease checks) across a full matrix of - (hunt_id, interval, now, watermark, lease), and the SQL implements that - predicate against real ClickHouse - counting exactly the runnable hunts, - excluding tombstoned / future-watermark / leased hunts, and returning 0 when - every hunt is caught up. -2. Single source of truth for the offset. The materialiser computes - `phase_offset` with the same `schedule.phase_offset` the worker uses, so the two - can never drift. Change the offset algorithm once and both move together. -3. Gate on the signal, not a timer. KEDA polls the backlog query (via the - shim - a real readiness signal), it does not race a clock. Worst-case wake - latency is - `pollingInterval + pod-start`. For hunts (minute-plus intervals) that is - immaterial. This is the "gate on the real condition" rule from the testing - standard applied to autoscaling. -4. Self-healing. If a worker claims a hunt then dies before writing the watermark, its lease expires (`lease_seconds`, default 300) and the fire is still owed (`watermark < fire`), so the scaler keeps >=1 worker and the hunt is reclaimed. Nothing double-runs (a hunt run twice writes and notifies twice): a claim stands only if acknowledged within a settle of its free read and still the latest row after it. A later claim is never released, as it may sit over the winner, but it holds nothing, so the next poll claims again. -5. No thundering herd. The phase offset spreads same-interval hunts across the - interval, so `due_count` rises gradually and KEDA scales out smoothly rather - than all-at-once at the boundary. The offset falls in the first - `hunts.phase_spread_fraction` of the interval (default 0.8, env - `DFE_ENGINE_HUNTS_PHASE_SPREAD_FRACTION`). The runner, the API's `next_due` and - the materialised schedule all read it, so republish the schedule - (`dfe-hunt-runner materialise`) after changing it. - -## Bounded catch-up - -A fire stays owed until its watermark is written, so a worker -that ticks inside `[boundary, boundary+offset)` runs the previous interval's fire -if nothing ran it, and the scaler counts it the same way. Without that, a hunt -whose offset sits between two poll ticks (a 46s offset polled at :00/:15/:30/:45) -never fires at all. Max lateness for any fire is under one interval - inherent -to the phase-offset design, and identical whether the decision is made by KEDA -or a resident worker. +A count above 0 scales up and 0 scales down, to zero only where `minReplicaCount: 0` is set (the default floor is 1 pod). The `enabled=1` filter sits in an inner subquery so a tombstone (interval 0) never reaches the `intDiv`. + +The watermark comparison is exact because the worker writes `watermark = scheduled_start`, the fire time, after a run commits (`worker.py`, `set_watermark(end)`). The watermark is in the same scale as the boundary arithmetic, so `watermark < fire` means "this fire has not been completed". + +## Why KEDA and a worker agree + +Both compute "due" from the same inputs: + +1. The backlog SQL and `runner.tick`'s gate share `schedule.latest_fire`, the watermark and the lease check. A parity test runs a matrix of (hunt_id, interval, now, watermark, lease), and the SQL runs against real ClickHouse. +2. The materialiser writes `phase_offset` with the same `schedule.phase_offset` the worker uses. Change the offset algorithm once and both move. +3. KEDA polls a real signal. Worst-case wake latency is `pollingInterval` plus pod start, which is immaterial for minute-plus intervals. +4. A worker that claims a hunt then dies before writing the watermark loses its lease after `lease_seconds` (default 300). The fire is still owed, so the backlog stays above 0 and another worker reclaims it. +5. Nothing double-runs. A claim stands only if it is acknowledged within a settle of its free read and is still the latest row after it. A later claim is never released, because it may sit over the winner, but it holds nothing, so the next poll claims again. +6. Hunts do not all fire at once. The phase offset falls in the first `hunts.phase_spread_fraction` of the interval (default 0.8, `DFE_ENGINE_HUNTS_PHASE_SPREAD_FRACTION`), so `due_count` rises gradually and KEDA scales out smoothly. The runner, the API's `next_due` and the materialised schedule all read it, so republish the schedule (`dfe-hunt-runner materialise`) after changing it. + +A fire stays owed until its watermark is written. A worker that ticks inside `[boundary, boundary+offset)` runs the previous interval's fire if nothing ran it, and the backlog counts it the same way. Without that, a hunt whose offset sits between two poll ticks (a 46s offset polled at :00/:15/:30/:45) never fires. Maximum lateness for any fire is under one interval, whether KEDA or a resident worker makes the decision. ## No operational dependency on dfe-engine -This is the point of the pull-based design. Once deployed, the running system needs -only ClickHouse: - -- Workers read hunt defs from the gitops config (mounted / cloned), coordinate - through the coordination tables, and execute queries against CH. No engine call on the - hot path. -- KEDA reads `hunt_schedule` + the coordination tables from CH via the - `dfe-keda-shim` - a separate, distinctly-labelled pod running the SAME - engine container image under a different entrypoint (`dfe-keda-shim run`, - not the live `dfe-engine` API service). No call to the running engine API. -- The engine's only role is at CONFIG-DEPLOY time, not runtime: materialise - `hunt_schedule` when the hunt config changes, with the one-shot, idempotent - `dfe-hunt-runner materialise`. The charts do not run it (see below). - -So if the engine pod is down, hunts keep firing, and so does KEDA scaling once -the schedule is materialised. The only -thing you lose while the engine is down is the ability to CHANGE configs - which -needs the engine anyway. ClickHouse is DFE's single operational store, and the runner -holds to that (no Postgres, no engine, no convenience store on the hot path), which -is also what lets the same runner work on a non-k8s single deploy (dfe-docker) where -only ClickHouse is guaranteed present. - -### Coordination needs a SINGLE LOGICAL ClickHouse (scale-tier requirement) - -The coordination tables are the shared source of truth every worker and KEDA -read and write. That only works if all of them see the SAME tables. On the **scale -tier's multi-node cluster behind a round-robin load balancer**, a plain -`ReplacingMergeTree` (and a plain `CREATE DATABASE`) would land on one node, so -workers whose connections land on other nodes would see an empty or absent table and -coordination would silently break (`UNKNOWN_DATABASE`, missing leases). The schema -phase therefore creates the coordination tables as `ReplicatedReplacingMergeTree` -inside the Replicated `dfe` database on every deployment, so lease, watermark and -schedule state is shared across nodes (). - -### Materialisation (config-deploy time) - -`schedule.publish_schedule(ch, db, specs)` is the ONLY writer of `hunt_schedule`, and -`dfe-hunt-runner materialise` is its only caller. It is idempotent and diff-aware -(tombstones hunts that vanished from config). The charts run no Job for it, -so on Kubernetes the table stays empty unless an operator runs the command after -each hunt-config change. Run that way, a brand-new hunt added while scaled to zero -still gets a `hunt_schedule` row, so KEDA can wake a worker for it. - -## KEDA ScaledObject (how it is wired) - -**KEDA has no native ClickHouse scaler.** The earlier approach - KEDA's `mysql` -scaler pointed at ClickHouse's MySQL-compatible interface (`mysql_port`, 9004) - -has been **RETIRED**: KEDA's only failure knob for a DB scaler is a fixed -`fallbackReplicas` count, not "hold at current", so a broken metric could scale -the wrong way and blow cloud spend. It is replaced by a small OWNED, fail-safe -adapter: the **`dfe-keda-shim`**, its own pod on the engine image (`dfe-keda-shim -run`), labelled so the engine Service never routes to it. The published engine -chart (`oci://ghcr.io/hyperi-io/charts/dfe-engine`) renders only the engine pod. The -shim comes from `dfe-infra helm/charts/dfe-extras/templates/engine-keda-shim.yaml`, -on the same values. KEDA's `metrics-api` scaler polls the shim over HTTP, and the -shim runs `schedule.due_query(db)` against ClickHouse - the SAME SQL the worker -gate uses (see "The scaler query" above, imported directly from -`dfe_engine.hunts.schedule.due_query`, so it cannot drift) - and hands -back one integer. +Once deployed, the running system needs only ClickHouse: + +- Workers read hunt definitions from the gitops config, coordinate through the coordination tables and run queries against ClickHouse. +- The shim reads `hunt_schedule` and the coordination tables from ClickHouse. It is a separate, distinctly labelled pod on the engine container image with its own entrypoint (`dfe-keda-shim`), not the live engine API. +- `hunt_schedule` is written at config-deploy time by `dfe-hunt-runner materialise`, a one-shot, idempotent command that rewrites it from the hunt config. The charts do not run it. + +If the engine pod is down, running hunt-runner pods keep firing hunts. What you lose is the ability to change configs, which needs the engine anyway. A pod that is starting is the exception: the runner and the shim each have a `wait-for-engine` init container (dfe-infra `helm/library/dfe-common/templates/_waitfor.tpl`) that holds the pod until the engine reports ready, because the engine applies the ClickHouse schema at its own start. A scale-out or a restart while the engine is down waits for it. ClickHouse is DFE's single operational store and the runner holds to that, which is also why the same runner works on a non-k8s single deploy (dfe-docker), where only ClickHouse is guaranteed present. + +Every worker and the shim must see the same coordination tables. On the scale tier's multi-node cluster behind a round-robin load balancer, a plain `ReplacingMergeTree` lands on one node, and workers connected to other nodes see an empty or absent table (`UNKNOWN_DATABASE`, missing leases). The schema phase therefore creates the coordination tables as `ReplicatedReplacingMergeTree` inside a Replicated database on every deployment, so lease, watermark and schedule state is shared across nodes (). + +`schedule.publish_schedule(ch, db, specs)` is the only writer of `hunt_schedule`, and `dfe-hunt-runner materialise` is its only caller. It is idempotent and diff-aware: it tombstones hunts that vanished from config. On Kubernetes the table stays empty until an operator runs the command after each hunt-config change. A hunt added while scaled to zero then still gets a row, so KEDA can wake a worker for it. + +## The ScaledObject + +KEDA has no native ClickHouse scaler, and a KEDA database scaler can only fall back to a fixed replica count, not hold the current one. So the hunt runner scales through the `dfe-keda-shim`, a small pod on the engine image that KEDA's `metrics-api` scaler polls over HTTP. The shim runs `schedule.due_query(db)`, the same function the worker gate uses, and returns one integer. + +The published engine chart (`oci://ghcr.io/hyperi-io/charts/dfe-engine`) renders only the engine pod. The shim and the hunt runner come from dfe-infra's `helm/charts/dfe-extras` chart (`templates/engine-keda-shim.yaml`, `templates/engine-hunt-runner.yaml`), on the same values. ```mermaid flowchart LR subgraph CH["ClickHouse"] hs[("hunt_schedule +\nhunt_lease/watermark/state")] end - shim["dfe-keda-shim\n(dfe-keda-shim run)\nGET /keda/hunt-backlog"] + shim["dfe-keda-shim\nGET /keda/hunt-backlog"] ko["KEDA metrics-api trigger\n(ScaledObject: dfe-hunt-runner-scaler)"] hr["dfe-hunt-runner Deployment\n(replicas: N)"] hs -->|"due_query(db)"| shim @@ -198,11 +89,7 @@ flowchart LR shim -. "on failure: last-good\ncache / cold-hold" .-> ko ``` -Ships with the hunt-runner Deployment, gated by `huntRunner.autoscaling.enabled` -(default `false`, **BETA**). When enabled, the Deployment's `replicas` field is -OMITTED (the ScaledObject owns the count instead) and a `dfe-hunt-runner-scaler` -`ScaledObject` is rendered alongside it -(`dfe-infra helm/charts/dfe-extras/templates/engine-hunt-runner.yaml`): +With `huntRunner.autoscaling.enabled`, the Deployment's `replicas` field is omitted, because the ScaledObject owns the count, and a `dfe-hunt-runner-scaler` ScaledObject is rendered beside it: ```yaml apiVersion: keda.sh/v1alpha1 @@ -226,67 +113,20 @@ spec: valueLocation: "value" ``` -## The shim's fail-safe - -The fail-safe (`src/dfe_engine/keda_shim/shim.py`, `QueryShim.run`): every query -caches its last-good integer per (query, params). On ANY failure - CH -unreachable, timeout, a bad query - the shim returns the CACHED value, so KEDA -sees no change and the pod count FREEZES at current, so it can never run -replicas up on a broken metric. A cold start with no cache yet, plus a -failure, falls back to a conservative `cold_hold` (0 for the backlog query - -hold at `minReplicaCount`). This cache-or-cold-hold behaviour is exactly why -the shim replaced the zero-code MySQL-on-ClickHouse route: KEDA's native DB -scalers cannot express "hold at current", only a fixed fallback count. - -Other notes: the shim is a separate, distinctly-labelled pod -(`app.kubernetes.io/name: dfe-keda-shim`) running the same engine container -image under a different entrypoint - it is NOT the live engine API, so -hunt-runner scaling still has no dependency on the engine service being up. -Queries (including the backlog query's clamp/cold-hold/timeout) are -config-driven (`src/dfe_engine/keda_shim/queries.yaml` + an optional mounted override), so a -schema change or a tuning change is a config edit, not a shim rebuild. - -### Proving it: the artificial scale test - -`bootstrap/keda-scale-test.sh` (dfe-infra) is the mechanised proof that the -shim -> KEDA scale-out/scale-in path actually works, rather than an -aspirational claim: it forces the path on demand with a disposable target -instead of waiting for real hunt load - stand up a throwaway Deployment + -ScaledObject wired to the shim's pressure query for a test `ServiceName`, -INSERT a high `dfe_scaling_pressure` row into `otel_metrics_gauge` (any -`_scaling_pressure` spelling matches - see -[keda-scaling](../deployment/keda-scaling.md)), poll -(bounded, 120s) for scale-OUT to >= 2 replicas, drag the rolling average back -down, poll (bounded, 180s) for scale-IN to 1, then self-clean. It rides the -SAME shim endpoint and metrics-api mechanism the hunt-runner's -`/keda/hunt-backlog` trigger uses, so a pass proves the mechanism the -hunt-runner beta relies on, not a mock. Wired into -`bootstrap/run-all-smoke-tests.sh` (gated `DFE_KEDA_SCALE_TEST`, default on). - -## Default deployment, beta autoscaling, and disable - -The hunt runner is a thin orchestrator - virtually all hunt load runs IN -ClickHouse (the queries + coordination), so the pod itself is rarely -resource-bound. Its real ceiling is architectural (how many hunts one pod can -submit and coordinate concurrently and reliably), not CPU/memory - and the -large majority of deployments need exactly one pod. So the default is -deliberately simple: - -- `huntRunner.enabled` (default `true` on every tier): deploys `dfe-hunt-runner` as ONE steady pod - (`replicas: 1`, no ScaledObject at all). This is the resting state for the - large majority of deployments - a single always-on pod, not scale-to-zero. -- `huntRunner.autoscaling.enabled` (default `false`, BETA, keep it off, - see the top of this page): opts into the ScaledObject described above. `replicas` is then omitted from the - Deployment (KEDA owns the count) and the pod count floats between - `minReplicaCount` (default **1**, not 0 - set it to `0` explicitly via - values to additionally opt into scale-to-zero) and `maxReplicaCount` - (default 5), driven by the due backlog. Work still divvies up safely across - N pods via the existing `hunt_lease` mutual exclusion - KEDA picks the pod - count, the leases decide who does what, so nothing double-runs. -- Disabled (operator choice): set `huntRunner.enabled: false` in the - deploy overlay. No pod and no ScaledObject are rendered, so hunts do not fire. - Re-enable by restoring the dial - the schedule + watermarks are still in - ClickHouse, so the runner resumes cleanly from the last committed window. - -Either way the mechanism is a values dial, turned through the gitops overlay, -never a live cluster mutation. +## When the metric is missing + +The shim caches its last-good integer per query and parameters (`src/dfe_engine/keda_shim/shim.py`, `QueryShim.run`). On any failure (ClickHouse unreachable, a timeout, a bad query) it returns the cached value. KEDA sees no change, the pod count freezes at current, and a broken metric cannot run replicas up. A cold start with no cache yet, plus a failure, falls back to `cold_hold`, which is 0 for the backlog query. That holds the pod count at `minReplicaCount`. + +The queries (clamp, `cold_hold` and timeout included) are config in `src/dfe_engine/keda_shim/queries.yaml`, with an optional mounted override at `DFE_ENGINE_KEDA_SHIM_QUERY_CONFIG`. A schema change or a tuning change is a config edit, not a shim rebuild. + +`bootstrap/keda-scale-test.sh` in dfe-infra tests the shim to KEDA scale-out and scale-in path on demand with a disposable target. It drives the shim's pressure query, not the backlog query, but the metrics-api mechanism is the same one the backlog trigger uses. See [keda-scaling](../deployment/keda-scaling.md). + +## Values + +The defaults are in dfe-infra `helm/charts/dfe-extras/values.yaml`. + +- `huntRunner.enabled` (default `true`) deploys `dfe-hunt-runner` as ONE steady pod (`replicas: 1`, no ScaledObject). +- `huntRunner.autoscaling.enabled` (default `false`, beta) renders the ScaledObject above. The pod count floats between `minReplicaCount` (default 1, set `0` to opt into scale-to-zero) and `maxReplicaCount` (default 5). Work divides safely across N pods through the `hunt_lease` mutual exclusion: KEDA picks the pod count and the leases decide who does what. +- `huntRunner.enabled: false` in the deploy overlay renders no pod and no ScaledObject, so hunts do not fire. Re-enable it and the runner resumes from the last committed window, because the schedule and watermarks are still in ClickHouse. + +Each is a values dial, turned through the gitops overlay, never a live cluster mutation. diff --git a/docs/data-plane/hunt-schedule-smoothing.md b/docs/data-plane/hunt-schedule-smoothing.md index 8865a977..9048d16c 100644 --- a/docs/data-plane/hunt-schedule-smoothing.md +++ b/docs/data-plane/hunt-schedule-smoothing.md @@ -6,115 +6,94 @@ Copyright: (c) 2026 HYPERI PTY LIMITED --> -# Hunt schedule smoothing - flattening CH load by phase drift +# Hunt schedule smoothing -How DFE keeps a fleet of recurring detection queries from stacking up into CPU and -memory peaks on a shared ClickHouse cluster, by learning the load each one imposes -and drifting *when* (not how often) it fires so the aggregate curves flatten over -time. This is a closed-loop **remediation** approach, not admission control: we do -not gate or defer firing - we nudge phases and let the fleet settle. +Smoothing is held back from 2.2.2 until it has scaled testing, so no setting turns it on in this release. It comes in 2.2.3 as an opt-in beta behind `hunts.smoothing.enabled`, off by default. Tracked in https://github.com/hyperi-io/dfe-engine/issues/864. -> **Status: DESIGN -- not built, not an option in this release.** The adaptive loop on -> this page does not exist in the engine. There is no `smoothing.enabled` setting and -> nothing drifts phases. What ships is the static part: each rate hunt fires at a fixed -> phase offset hashed from its id (`hunts/schedule.py`), that offset is written to -> `hunt_schedule` for the KEDA due count, and every hunt query carries a JSON -> `log_comment` that `query_log_archive` attributes back to the hunt. +What 2.2.2 ships is the fixed offset: each rate hunt fires at an offset hashed from its id, within the first `hunts.phase_spread_fraction` of its interval (default 0.8, see [hunt-runner-scaling.md](hunt-runner-scaling.md#why-keda-and-a-worker-agree)). Every hunt query is also tagged with a JSON `log_comment`, so ClickHouse attributes its cost to the hunt. -## The picture (intuition first) +Rate hunts (`schedule.mode: rate`, the default) fire on fixed intervals. Left alone, the heavy ones stack into ClickHouse CPU and memory peaks. Smoothing learns what each hunt costs and drifts WHEN it fires, never how often, so the sum flattens over time. Anchored hunts and one-off monsters are meant to spike, so the loop leaves them alone. -The whole job is one change in the load curve. On the left, a fleet of recurring hunts that all fire -at the top of their period, so their cost pulses STACK into a few tall peaks on the shared -ClickHouse, with idle troughs between. On the right, after the loop drifts each hunt's -phase (WHEN it fires, never how often), the same total work SPREADS into the troughs and -the aggregate curve FLATTENS. Same coverage, same freshness - just no synchronised pile-ups. +Still design only, not in the 2.2.3 beta: the ClickHouse `WORKLOAD 'hunts'` backstop, promoting a settled schedule into git, and auto-disable. -```text -Aggregate ClickHouse pressure P(b) across one hyperperiod H (b = one Delta-second bucket) - - BEFORE - all fire on the period boundary AFTER - phases drifted apart - - P | # P | - | # | x x x x x - | # # | x # x # x # x # x # x - | # # # | # # # # # # # # # # # - | x # x x # x x # x | # # # # # # # # # # # - +------------------------------> b +------------------------------> b - peak/mean ~ 4 (spiky: risks the peak/mean ~ 1.2 (flat: headroom - memory cliff) under the memory cap) -``` +## The picture + +Hunts whose hash offsets land close together stack their cost pulses. The loop drifts them apart: same coverage, same freshness, no pile-up. + +![Aggregate ClickHouse pressure P(b) per bucket across one hyperperiod H. Before smoothing three tall spikes stand over a low floor, peak over mean 4.6. After, the bars sit close to the mean, peak over mean 1.2.](images/hunt-smoothing-before-after.svg) + +The one number to watch is `dfe_hunt_smoothing_flatness`, peak over mean, falling toward 1.0. + +## Why memory gets special treatment + +CPU and memory fail differently. A CPU peak slows queries. A memory peak crosses the server limit and ClickHouse throws `MEMORY_LIMIT_EXCEEDED` (code 241) at whatever is allocating right then, so one spike fails many co-running queries, innocent ones included. A slope and a cliff. + +![Left, CPU: p95 latency rises on a soft slope as load grows, so queries just slow down. Right, memory: query success stays flat until the server memory limit, then drops at once as code 241 fails many queries.](images/hunt-smoothing-cpu-vs-memory.svg) + +So the loop weights both, and memory is also a hard constraint: no move may push any bucket's summed memory over the cap. -The one number that tracks this is `dfe_hunt_smoothing_flatness` = peak/mean, falling -toward 1.0 as the curve flattens; the objective J (in the maths section) is just the -formal handle on "make this flat". If you read one section, read this one - the maths -only proves the drift provably lowers those peaks and settles. - -## Scope - -- **In scope: rate hunts (loop-queries)** - the "run every N minutes/hours" - detections (see [HUNT-RUNNER-SCALING.md](hunt-runner-scaling.md), `mode: rate`). - These get phase-offset spreading and the adaptive smoothing below. -- **Out of scope: anchored hunts and monster one-offs.** The deliberately-spiky - `mode: anchored` runs (e.g. a new TI checked against 270 days of history at a - weekend lull) are *meant* to spike, and sizing the cluster for them is the infra - team's problem, not the scheduler's. The smoother ignores them (beyond keeping - their memory off the cliff via the guard below). - -## The two drivers (this is huge OLAP, so both matter, differently) - -A recurring query stresses two resources with two very different failure modes: - -- **CPU** - heavy calculation / transform / aggregation. A CPU peak makes queries - *slow*. Soft and recoverable. -- **Memory** - big scans, joins, large `GROUP BY`. When several land together and - cross the server memory limit, ClickHouse throws `MEMORY_LIMIT_EXCEEDED` (code 241) - at whatever is allocating right then - so a memory spike **fails in-flight queries, - innocent bystanders included**, not just the offenders. That is an availability - cliff, not a node death: with sane limits (`max_server_memory_usage` leaving OS - headroom + per-query `max_memory_usage`) ClickHouse sheds queries and STAYS UP - - shedding queries is exactly what the server limit is for. A true OS/cgroup OOM-KILL - of the CH process (a crash/restart) is a separate, catastrophic mode, but it takes - a misconfiguration (no OS headroom, wrong cgroup detection, limits above real RAM), - not concurrent heavy queries alone. - -So the smoother optimises a weighted composite of both, but memory still gets extra -protection: its failure is a cliff (a spike fails *many* co-running queries at once), -not a slope (CPU just slows), so memory is a hard constraint backed by CH's per-query -`max_memory_usage` guard (below). - -Visually the two failure modes could not be more different - which is why memory is the -hard constraint and CPU is only weighted: +## How it works + +1. The operator's interval is untouched. Only the offset into it moves. +2. Every hunt query carries a JSON `log_comment` naming the hunt and its fire. ClickHouse records that in `system.query_log`, and the `query_log_archive` view keeps it with the real cost columns. +3. One runner replica leads. Each cycle it reads every hunt's trailing p95 cost from the archive, sums the pulses into a load curve over the hyperperiod, finds the tallest bucket, and moves ONE hunt that fires into it by a bounded step toward the deepest trough. A move is kept only if it lowers the objective by more than a hysteresis margin, and a hunt just moved sits out a cool-down. +4. The new offset is written to `hunt_schedule`, and in the beta the runner, the API's next-due and the KEDA due count all read that one table, so every reader agrees on the next fire. In 2.2.2 only the KEDA due count reads it, and the runner and the API compute the hash offset themselves. + +Moving an offset changes one window's length and nothing else. A hunt moved later runs one longer window. A hunt moved earlier runs one shorter window. Coverage stays contiguous because every window starts at the last watermark. + +With the flag off, nothing reads the archive, nothing leads, and every offset is the hash. Turning the flag off after a run rewrites `hunt_schedule` back to the hash offsets on the runner's next start. + +Rescheduling fixes contention and nothing else. A query that busts the memory limit on its own busts it at 3am too, and moving it changes nothing. The ladder for that is still design only: smooth, then back off the interval of a hunt that runs into itself, then auto-disable a hunt that trips code 241 K times in a window, written as a governed-ops `enabled: false` with the reason, alerted, and never re-enabled without a human. + +## What a hunt costs + +Per hunt, per fire, over a trailing window (`hunts.smoothing.window_days`): + +$$ +m_i = \operatorname{p95}_{\text{fires}}\bigl(\max \texttt{memory\_usage}\bigr), \qquad +d_i = \operatorname{p95}_{\text{fires}}\Bigl(\sum \texttt{query\_duration\_ms}\Bigr), \qquad +c_i = d_i +$$ ```text - CPU (soft slope: it just slows) MEMORY (hard cliff: sheds many at once) - - p95 latency query success - | _.-' |--------------. - | _.-' | | - | _.-' | | - |_.-' | '-------- code 241 - +------------------> load +------------------> load - degrades gently, recoverable fine ... fine ... then a spike fails - the co-runners allocating right then +for each hunt i, over its fires in the trailing window: + m_i = p95 of each fire's max(memory_usage) the fire's peak memory + d_i = p95 of each fire's sum(query_duration_ms) the fire's duration, in buckets + c_i = d_i CPU, proxied by duration ``` -## The mechanism (remediation loop, non-destructive) +`query_log_archive` has no CPU column, so duration stands in for CPU. A hunt with no rows in the window stays on its current offset. A missing archive leaves every hunt where it is and logs once. -1. Fire what the operator asked for (their cron/interval - untouched). -2. Tag and observe each query's real cost over time from ClickHouse. -3. Fix overlaps, then incrementally drift phases so the CPU and memory curves - flatten. -4. Commit the adjusted phases to the gitops repo (auditable); the runners reconcile - on their next config reload (no restart - it is pull-based, nothing is pinned to - a pod). -5. **Settling:** once a hunt's schedule has stopped moving for a configured period, - promote the settled values *over* the operator's original guess in git (default - ON - see Config). +On ClickHouse Cloud `system.query_log` is per node, and a server upgrade can rename it under the view (ClickHouse issue 124177, fixed in 124217). A view that has stopped filling shows as every hunt's cost going stale, and the loop stops moving anything rather than move on old numbers. -Phase drift is deliberately non-destructive: it changes only *when within its -period* a hunt fires, never how often, so coverage and freshness are unchanged. The -phase offset was a hash guess to begin with, never operator intent - so learning a -better one and eventually committing it loses nothing. +## Settings + +The 2.2.3 beta's settings. None of them exists in 2.2.2. + +| Setting | Default | Meaning | +|---|---|---| +| `hunts.smoothing.enabled` | false | the master switch | +| `hunts.smoothing.cycle_seconds` | 300 | how often the leader runs a cycle | +| `hunts.smoothing.bucket_seconds` | 30 | Delta, the load curve's resolution | +| `hunts.smoothing.window_days` | 7 | the trailing window the p95 costs come from | +| `hunts.smoothing.max_step_fraction` | 0.1 | the most an offset moves per cycle, as a share of its interval | +| `hunts.smoothing.hysteresis` | 0.01 | the least relative drop in the objective that keeps a move | +| `hunts.smoothing.cooldown_cycles` | 2 | cycles a moved hunt sits out | +| `hunts.smoothing.weight_memory` | 0.5 | w_m, with w_c = 1 - w_m | +| `hunts.smoothing.memory_cap_bytes` | 0 | M_cap for the hard constraint, 0 reads `max_server_memory_usage` | +| `smoothing.settling_period`, `autodisable.*` | - | design only, see the top of this page | + +Environment form: `DFE_ENGINE_HUNTS_SMOOTHING_`. + +## Metrics + +The beta's leading runner emits these each cycle, through scalo's OTel exporter ([observability-standard.md](../deployment/observability-standard.md)): + +- `dfe_hunt_smoothing_enabled` (every runner, 1 when the flag is on), `dfe_hunt_smoothing_flatness` (peak/mean of P), `dfe_hunt_smoothing_objective` (J). +- `dfe_hunt_cost_mem_bytes{hunt}`, `dfe_hunt_cost_duration_ms{hunt}`. +- `dfe_hunt_phase_offset{hunt}`, `dfe_hunt_phase_shift_total{hunt}`. + +J trending down is the loop working. J low and flat is settled. The runner's own `dfe_hunt_backlog`, `dfe_hunt_runs_total` and `dfe_hunt_overruns_total` ship in 2.2.2. --- @@ -124,56 +103,91 @@ better one and eventually committing it loses nothing. ### Setup -Index the rate hunts i = 1..N. Each has period T_i (seconds) and a phase offset -φ_i ∈ [0, T_i) - the free variable we control. The **hyperperiod** is -H = lcm(T_1, ..., T_N); for the usual harmonic periods (300, 900, 3600 s) that is -just 3600 s. Discretise H into B buckets of width Δ = H / B (e.g. Δ = 30 s). Within -H, hunt i fires H / T_i times, at bucket positions ⌊(φ_i + k·T_i) / Δ⌋ mod B for -k = 0, 1, .... +Index the rate hunts $i = 1, \dots, N$. Each has period $T_i$ (seconds) and a phase offset +$\varphi_i \in [0, T_i)$ - the free variable we control. The **hyperperiod** is +$H = \operatorname{lcm}(T_1, \dots, T_N)$; for the usual harmonic periods (300, 900, 3600 s) that is +just 3600 s. Discretise $H$ into $B$ buckets of width $\Delta = H / B$ (e.g. $\Delta = 30$ s). Within +$H$, hunt $i$ fires $H / T_i$ times, at bucket positions $\bigl\lfloor (\varphi_i + k T_i) / \Delta \bigr\rfloor \bmod B$ for +$k = 0, 1, \dots$. -From `system.query_log` (per hunt, p95) we learn a **cost pulse**: a duration d_i -buckets wide carrying a CPU magnitude c_i and a peak-memory magnitude m_i. A fire at -offset φ deposits that pulse across the d_i buckets it covers (wrapping around H). +From `system.query_log` (per hunt, p95) we learn a **cost pulse**: a duration $d_i$ +buckets wide carrying a CPU magnitude $c_i$ and a peak-memory magnitude $m_i$. A fire at +offset $\varphi$ deposits that pulse across the $d_i$ buckets it covers (wrapping around $H$). ### Aggregate curves -For each bucket b = 0..B-1, summing every fire of every hunt that covers b: +For each bucket $b = 0, \dots, B-1$, summing every fire of every hunt that covers $b$: + +$$ +C(b) = \sum_i \sum_k c_i \cdot \mathbb{1}\bigl[\text{fire } (i,k) \text{ covers } b\bigr] \quad \text{(CPU)} +\qquad +M(b) = \sum_i \sum_k m_i \cdot \mathbb{1}\bigl[\text{fire } (i,k) \text{ covers } b\bigr] \quad \text{(memory)} +$$ ```text -C(b) = Σ_i Σ_k c_i · 𝟙[fire (i,k) covers b] (CPU) -M(b) = Σ_i Σ_k m_i · 𝟙[fire (i,k) covers b] (memory) +C = [0] * B +M = [0] * B +for each hunt i: + for k in 0 .. H/T_i - 1: # every fire of hunt i in the hyperperiod + first = floor((phi_i + k * T_i) / Delta) + for j in 0 .. d_i - 1: # the d_i buckets the fire covers + b = (first + j) mod B # wrap around H + C[b] += c_i + M[b] += m_i ``` -Normalise each by cluster capacity: c̃(b) = C(b) / C_cap, m̃(b) = M(b) / M_cap. -The **composite pressure** in bucket b, with weights w_c + w_m = 1: +Normalise each by cluster capacity: $\tilde{c}(b) = C(b) / C_{\text{cap}}$, $\tilde{m}(b) = M(b) / M_{\text{cap}}$. +The **composite pressure** in bucket $b$, with weights $w_c + w_m = 1$: + +$$ +P(b) = w_c \, \tilde{c}(b) + w_m \, \tilde{m}(b) +$$ ```text -P(b) = w_c · c̃(b) + w_m · m̃(b) +for b in 0 .. B-1: + P[b] = w_c * C[b] / C_cap + w_m * M[b] / M_cap ``` ### Objective Minimise the sum of squares of the composite pressure over the hyperperiod: +$$ +\begin{aligned} +\text{minimise} \quad & J(\varphi) = \sum_{b=0}^{B-1} P(b)^2 \\ +\text{subject to} \quad & \varphi_i \in [0, T_i) \quad \text{for all } i && \text{(full-period slack)} \\ +& M(b) \le M_{\text{cap}} \quad \text{for all } b && \text{(keep concurrent memory under the server limit)} +\end{aligned} +$$ + ```text -minimise J(φ) = Σ_{b=0}^{B-1} P(b)² -subject to φ_i ∈ [0, T_i) for all i (full-period slack) - M(b) ≤ M_cap for all b (keep concurrent memory under the server limit) +J(phi) = sum(P[b] ** 2 for b in 0 .. B-1) +a schedule phi is allowed only if + 0 <= phi_i < T_i for every hunt i + M[b] <= M_cap for every bucket b +pick the allowed phi with the smallest J ``` **Why sum-of-squares is the flattening objective.** The total pressure -T = Σ_b P(b) is *invariant* under phase changes - moving φ_i only shifts where a -hunt's pulses land, not their total (H is periodic, so pulses wrap, nothing falls +$T = \sum_b P(b)$ is *invariant* under phase changes - moving $\varphi_i$ only shifts where a +hunt's pulses land, not their total ($H$ is periodic, so pulses wrap, nothing falls off the edge). Since +$$ +\sum_b P(b)^2 = \frac{T^2}{B} + \sum_b \bigl(P(b) - \bar{P}\bigr)^2 = \text{const} + B \cdot \operatorname{Var}(P), +$$ + ```text -Σ_b P(b)² = T²/B + Σ_b (P(b) − P̄)² = const + B · Var(P), +T = sum(P) # the same for every phi +P_bar = T / B # the mean bucket +J = T**2 / B + sum((P[b] - P_bar) ** 2 for b in 0 .. B-1) + = const + B * Var(P) # the only part phi can change is the variance ``` minimising J is *exactly* minimising the variance of the load curve - i.e. -flattening it. And by Cauchy-Schwarz, J ≥ T²/B with equality iff P(b) is constant +flattening it. And by Cauchy-Schwarz, $J \ge T^2/B$ with equality iff $P(b)$ is constant (perfectly flat). This is the same objective the two source fields use (resource -smoothing's Σ R_t²; valley-filling's ℓ2-norm minimisation). +smoothing's $\sum_t R_t^2$; valley-filling's $\ell_2$-norm minimisation). ### The optimiser - damped, one-at-a-time water-filling @@ -193,172 +207,43 @@ makes a new peak). Instead, each control cycle: 7. Wait one cycle (let observation catch up); repeat. ``` -Moving one hunt at a time, by a bounded step, with a hysteresis margin θ and a +Moving one hunt at a time, by a bounded step, with a hysteresis margin $\theta$ and a per-hunt cool-down, is what makes it the gentle "drift" - and what prevents oscillation and avalanche. -**Convergence.** Every accepted move strictly decreases J, and J ≥ 0 is bounded -below, so J converges monotonically. This is coordinate descent on J; it reaches a +**Convergence.** Every accepted move strictly decreases $J$, and $J \ge 0$ is bounded +below, so $J$ converges monotonically. This is coordinate descent on $J$; it reaches a local minimum, which - as the resource-leveling heuristics find in practice - is where nearly all the benefit lives (the first few moves kill the worst peaks). ### Settling and promotion -Let the effective (learned) schedule for hunt i live in a gitcrud overlay separate -from the operator's committed def. Hunt i is **settled** when no move has been -accepted for it in a window ≥ X (the settling period). On settling, if promotion is -enabled (default), commit the settled (φ_i, and any backed-off T_i from the +Let the effective (learned) schedule for hunt $i$ live in a gitcrud overlay separate +from the operator's committed def. Hunt $i$ is **settled** when no move has been +accepted for it in a window $\ge X$ (the settling period). On settling, if promotion is +enabled (default), commit the settled ($\varphi_i$, and any backed-off $T_i$ from the runs-into-itself loop) *over* the operator's original values in git - with an audit -message - and clear i's overlay entry. Future drift then starts from the settled +message - and clear $i$'s overlay entry. Future drift then starts from the settled baseline. This keeps the committed config truthful (it reflects what actually runs well, not the initial hash guess) and bounds overlay growth. --- -## Cost model - -Attribution needs one cheap enabler, and it ships today: every hunt query carries a -JSON `log_comment` (`DfeQueryTags`: `feature=hunts`, `kind=hunt`, `id=`), which the -`query_log_archive` view keeps. No `workload` is set, because the `hunts` WORKLOAD is not -provisioned and naming an undefined one fails the query. Then per hunt, over a trailing -window: - -```text -c_i ≈ p95( ProfileEvents CPU µs ) from system.query_log (type = QueryFinish) -m_i ≈ p95( peak_memory_usage ) " -d_i ≈ p95( query_duration_ms ) / Δ " -``` - -Because it is a trailing p95 refreshed each cycle, the model tracks data drift -automatically - a table that grows or a join that gets heavier raises c_i/m_i -and the smoother re-drifts. This same telemetry feeds the interval-backoff loop -(runs-into-itself) described in HUNT-RUNNER-SCALING.md. - -## ClickHouse-native backstop (planned) - -Hunt queries set no `WORKLOAD` in this release. The planned backstop: the smoother -*optimises* the curve proactively but can still mis-predict when the data changes abruptly, -so hunt queries would also run inside a ClickHouse `WORKLOAD 'hunts'` with a weight -(fair-share against ingestion / UI) and, per query, `max_memory_usage`. That gives a -hard ceiling and reactive fair-share independent of our loop: our maths smooths the -load, CH guarantees no single hunt query exceeds its memory cap - so a bad one fails -on its own rather than starving co-runners or feeding an OS OOM-kill. - -## Metrics (scalo) - the controller is fully observable - -A control loop you cannot see is one you cannot trust. Every input the smoother -decides on, every change it makes, and everything it gives up on is emitted as a -scalo metric through the standard OTel exporter (see -[OBSERVABILITY-STANDARD.md](../deployment/observability-standard.md); default sink HyperDX). Three -buckets: - -**1. What it is running** (runner/daemon operational state): - -- These ship today: `dfe_hunt_backlog` (the due-but-unclaimed backlog, also the KEDA - signal), `dfe_hunt_claims_total`, `dfe_hunt_runs_total`, `dfe_hunt_overruns_total` - and `dfe_hunt_tick_duration_seconds`. - -**2. What it changes** (the actuations - so every adjustment is auditable in metrics -as well as in git): - -- `dfe_hunt_phase_shift_total{hunt}` + a `dfe_hunt_phase_offset{hunt}` gauge, -- `dfe_hunt_interval_backoff_total{hunt}` + `dfe_hunt_interval_seconds{hunt}`, -- `dfe_hunt_schedule_promotions_total{hunt}` (settling promotions), -- `dfe_hunt_autodisabled{hunt,reason}` (see below). - -**3. The combined/derived metrics it decides ON** - the novel bit: the controller's -own inputs and objective are metrics in their own right, so you can watch it work and prove -it converges: - -- per-hunt cost model (trailing p95): `dfe_hunt_cost_cpu_us{hunt}`, - `dfe_hunt_cost_mem_bytes{hunt}`, `dfe_hunt_cost_duration_ms{hunt}`, -- the aggregate curves: `dfe_hunt_load_cpu`, `dfe_hunt_load_mem`, - `dfe_hunt_load_pressure` (the composite P) - each as peak, mean, and peak/mean, -- the objective: `dfe_hunt_smoothing_objective` (J = Σ P²) and - `dfe_hunt_smoothing_flatness` (peak/mean, -> 1.0 as it flattens). J trending down - is the proof the loop is working; J low-and-flat is "settled". - -Failure metrics (they gate auto-disable): `dfe_hunt_mem_limit_total{hunt}`, -`dfe_hunt_mem_limit_crashloop_total{hunt}`, `dfe_hunt_threshold_breach_total{hunt,resource}`. - -## What cannot be fixed - escalation and auto-disable - -Rescheduling only fixes *contention*. A query that busts the memory limit will bust -it at 3am as surely as at peak - moving it changes nothing. Such hunts must be flagged and -stopped, not smoothed forever. The escalation ladder: - -1. **Smooth** (phase drift) - resolve co-scheduling peaks. Fixes contention. -2. **Back off** (interval) - give a slow hunt room when it runs into itself. Fixes a - hunt that is only slow. -3. **Auto-disable** - if it still fails, it is unfixable by scheduling: stop it and - flag a human. The terminal "cannot be fixed" state. - -Auto-disable triggers: - -- **Memory-limit crash-loop (default ON - a safety, not opt-in).** A hunt whose query - trips `MEMORY_LIMIT_EXCEEDED` (CH code 241) K times within a window is auto-disabled. - A query that keeps blowing the memory limit fails itself AND the co-running queries - allocating at that moment, so *retrying* it just repeats the collateral damage - we - stop it. This is the last line behind the per-query - `max_memory_usage` guard: the guard kills the one query; if it keeps happening, we - kill the schedule. -- **Consumption thresholds (opt-in, exposed by API).** An operator sets, per-hunt or - globally, a max peak-memory / max CPU / max duration; a breach auto-disables (or - warn-then-disable). Governed-ops config - set/get via the API, committed to git. - -Mechanics (consistent with the rest of the architecture): - -- Auto-disable is a **governed-ops action**: write `enabled: false` + reason + - timestamp to the hunt's state overlay in gitcrud (an auditable "why"), which the - runners reconcile and KEDA stops counting. It is a **circuit-breaker**, distinct - from the transient defer/overrun signal - defer self-heals; auto-disable stays open - until a **human** fixes the query (or raises the threshold) and re-enables via the - API. It never silently re-enables; the fault is not self-healing. -- Every auto-disable emits `dfe_hunt_autodisabled{hunt,reason}` AND raises an alert - - exactly what alerting should surface: a detection was killed and is no longer - running. - -## Config knobs - -Planned, not built: none of these settings exist in the engine today. - -| Knob | Default | Meaning | -|---|---|---| -| `smoothing.enabled` | off | master switch for the adaptive loop | -| `smoothing.weights` | `{cpu: 0.5, mem: 0.5}` | w_c / w_m in the composite P | -| `smoothing.bucket_seconds` | 30 | Δ, the curve resolution | -| `smoothing.max_step` | (period-relative) | Δφ_max, the per-cycle drift clamp | -| `smoothing.hysteresis` | small | θ, the min ΔJ to accept a move | -| `smoothing.settling_period` | on, X = 7 d | after X settled, promote learned schedule over the original in git | -| `smoothing.memory_cap` | from CH | M_cap for the hard constraint; also set CH `max_memory_usage` | -| `autodisable.mem_crashloop` | on, K=3 / 1 h | auto-disable a hunt that trips the memory limit K times in the window (safety, not opt-in) | -| `autodisable.max_memory` | off (opt-in) | per-hunt/global peak-memory ceiling -> disable on breach (API-set) | -| `autodisable.max_cpu` | off (opt-in) | CPU ceiling -> disable on breach (API-set) | -| `autodisable.max_duration` | off (opt-in) | duration ceiling -> disable on breach (API-set) | - -## Where it sits - -What ships is the static phase offsets and the JSON `log_comment` tagging. The offset -in `hunt_schedule` is read only by the KEDA due count: the runner, materialise and the -API's next-due each compute the hash offset themselves, so the smoother first needs one -offset source that all of them read, and materialise must stop overwriting learned -offsets. The cost model comes before that. - ## Useful Links and References -- **Resource smoothing** (operations research / project scheduling): shift tasks +- Resource smoothing (operations research / project scheduling): shift tasks within slack to flatten a resource histogram; classic objective Σ_t R_t², via right/left-shift heuristics and MINSLK - [PMI](https://www.pmi.org/learning/library/time-constrained-approach-resource-leveling-1737). -- **Valley-filling / demand response** (smart grid): flatten aggregate demand by +- Valley-filling / demand response (smart grid): flatten aggregate demand by minimising the ℓ2-norm; decentralised water-filling one load at a time against a shared signal, with the avalanche caveat - this one is pretty cool, sometimes a side study pays off in places you least expect - [decentralised EV charging](https://arxiv.org/pdf/1710.05533), [survey](https://arxiv.org/pdf/1911.06500). -- **Adaptive periodic spreading** (systems): APIO spreads periodic I/O and extends +- Adaptive periodic spreading (systems): APIO spreads periodic I/O and extends the period under contention (our interval-backoff) - [APIO](https://www.mdpi.com/2079-9292/11/9/1318). -- **ClickHouse-native** workload / CPU-slot scheduling + concurrency/memory caps - +- ClickHouse-native workload / CPU-slot scheduling + concurrency/memory caps - the reactive backstop, not the arrival-time shaper - [CH docs](https://clickhouse.com/docs/operations/workload-scheduling). diff --git a/docs/data-plane/images/hunt-smoothing-before-after.svg b/docs/data-plane/images/hunt-smoothing-before-after.svg new file mode 100644 index 00000000..1f8d52d9 --- /dev/null +++ b/docs/data-plane/images/hunt-smoothing-before-after.svg @@ -0,0 +1,506 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.2, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + bucket b across one hyperperiod H + + + + + pressure P(b) + + + + + + + + + + + + + mean + + + Before: offsets collide + peak/mean = 4.6 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + bucket b across one hyperperiod H + + + + + + + + + + + + + + mean + + + After: phases drifted apart + peak/mean = 1.2 + + + + + + + + + + + + diff --git a/docs/data-plane/images/hunt-smoothing-cpu-vs-memory.svg b/docs/data-plane/images/hunt-smoothing-cpu-vs-memory.svg new file mode 100644 index 00000000..a418a4a4 --- /dev/null +++ b/docs/data-plane/images/hunt-smoothing-cpu-vs-memory.svg @@ -0,0 +1,328 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.2, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + load + + + + + p95 latency + + + + + + + + + + + + + CPU: a soft slope, queries slow down + + + + + + + + + load + + + + + query success + + + + + + + + + + + + + + + + server memory limit + + + code 241 fails + many queries at once + + + Memory: a hard cliff, many fail at once + + + + + + + + + + + + diff --git a/docs/data-plane/images/hunt_smoothing_figures.py b/docs/data-plane/images/hunt_smoothing_figures.py new file mode 100644 index 00000000..7cad616c --- /dev/null +++ b/docs/data-plane/images/hunt_smoothing_figures.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +# Project: dfe-engine +# File: docs/data-plane/images/hunt_smoothing_figures.py +# Purpose: Draw the figures for docs/data-plane/hunt-schedule-smoothing.md +# Language: Python +# +# License: BUSL-1.1 +# Copyright: (c) 2026 HYPERI PTY LIMITED +"""Draw the hunt schedule smoothing figures as SVG, beside this script. + +The curves are illustrative shapes, not measurements. Each SVG has a transparent +background and mid-grey ink, so it reads on the docs site's light and dark themes. + +Usage:: + + uv run --with matplotlib python docs/data-plane/images/hunt_smoothing_figures.py +""" + +from pathlib import Path + +# matplotlib comes from `uv run --with`, not a project dependency, so ty cannot resolve it. +import matplotlib as mpl # ty: ignore + +mpl.use("svg") +import matplotlib.pyplot as plt # ty: ignore + +HERE = Path(__file__).resolve().parent +INK = "#85847f" +BLUE = "#3987e5" +ORANGE = "#e3602b" +BUCKETS = 24 + +plt.rcParams.update( + { + "svg.fonttype": "none", + "svg.hashsalt": "dfe-hunt-smoothing", + "font.family": "sans-serif", + "font.size": 11, + "text.color": INK, + "axes.edgecolor": INK, + "axes.labelcolor": INK, + "axes.titlecolor": INK, + "xtick.color": INK, + "ytick.color": INK, + "axes.spines.top": False, + "axes.spines.right": False, + "figure.facecolor": "none", + "axes.facecolor": "none", + "savefig.transparent": True, + } +) + + +def _before() -> list[float]: + """Pressure per bucket with three heavy hunts stacked on colliding offsets.""" + load = [1.0] * BUCKETS + for peak, height in ((3, 9.0), (10, 6.0), (17, 4.0)): + load[peak] += height + load[peak - 1] += height / 4 + load[peak + 1] += height / 4 + return load + + +def _after(total: float) -> list[float]: + """The same total pressure with the phases drifted apart, as a small zero-sum ripple.""" + mean = total / BUCKETS + ripple = (0.18, -0.06, -0.12, 0.0) + return [mean * (1 + ripple[b % len(ripple)]) for b in range(BUCKETS)] + + +def _bars(ax: plt.Axes, load: list[float], title: str) -> None: + """One panel of the before-and-after pair, with its mean and peak/mean marked.""" + mean = sum(load) / len(load) + ax.bar(range(len(load)), load, width=0.8, color=BLUE) + ax.axhline(mean, color=INK, linestyle="--", linewidth=1) + ax.text(len(load) - 0.6, mean, "mean", va="bottom", ha="right") + ax.set_title(f"{title}\npeak/mean = {max(load) / mean:.1f}", loc="left") + ax.set_xlabel("bucket b across one hyperperiod H") + ax.set_xticks([]) + + +def before_after() -> None: + """Aggregate ClickHouse pressure P(b), before and after smoothing.""" + before = _before() + after = _after(sum(before)) + fig, (left, right) = plt.subplots(1, 2, figsize=(10, 3.6), sharey=True) + _bars(left, before, "Before: offsets collide") + _bars(right, after, "After: phases drifted apart") + left.set_ylabel("pressure P(b)") + left.set_yticks([]) + fig.tight_layout() + fig.savefig(HERE / "hunt-smoothing-before-after.svg", metadata={"Date": None}) + plt.close(fig) + + +def cpu_vs_memory() -> None: + """CPU degrades on a slope, memory fails at a cliff.""" + load = [x / 100 for x in range(101)] + limit = 0.7 + fig, (cpu, mem) = plt.subplots(1, 2, figsize=(10, 3.4)) + + cpu.plot(load, [0.15 + 0.85 * x**2 for x in load], color=BLUE, linewidth=2) + cpu.set_title("CPU: a soft slope, queries slow down", loc="left") + cpu.set_ylabel("p95 latency") + + mem.plot(load, [1.0 if x < limit else 0.2 for x in load], color=ORANGE, linewidth=2) + mem.axvline(limit, color=INK, linestyle="--", linewidth=1) + mem.text(limit + 0.02, 0.62, "server memory limit") + mem.text(limit + 0.02, 0.3, "code 241 fails\nmany queries at once") + mem.set_title("Memory: a hard cliff, many fail at once", loc="left") + mem.set_ylabel("query success") + mem.set_ylim(0, 1.15) + + for ax in (cpu, mem): + ax.set_xlabel("load") + ax.set_xticks([]) + ax.set_yticks([]) + fig.tight_layout() + fig.savefig(HERE / "hunt-smoothing-cpu-vs-memory.svg", metadata={"Date": None}) + plt.close(fig) + + +if __name__ == "__main__": + before_after() + cpu_vs_memory() diff --git a/docs/data-plane/sigma.md b/docs/data-plane/sigma.md index 3c25e655..7d09c210 100644 --- a/docs/data-plane/sigma.md +++ b/docs/data-plane/sigma.md @@ -9,9 +9,11 @@ # Sigma detection pipeline -Sigma is off by default: every path under `/api/v1/sigma` answers 404 until the engine runs with `sigma.enabled` set, which is `DFE_ENGINE_SIGMA_ENABLED=true` in its environment. A `sigma` entry in a source's `views` is a field-map view and deploys either way. +Sigma is a beta, held back until it is tested from feed to detection. It is off by default behind `sigma.enabled` (`DFE_ENGINE_SIGMA_ENABLED=true`), and while it is off every path under `/api/v1/sigma` answers 404. The work to finish it is tracked in https://github.com/hyperi-io/dfe-engine/issues/887. -DFE turns community + commercial Sigma rules into live DFE detections. External +What works today is the source mapping. A `sigma` entry in a source's `views` is a field-map view, and it deploys whether the flag is on or off. The console does not offer Sigma for a new view, and an existing view keeps it. The rest of the pipeline sits behind the flag, works through the API only with no console flow, and has not yet run from feed to detection. + +The design turns community and commercial Sigma rules into DFE detections. External feeds are pulled into an operator-curated catalogue. The operator SELECTS the rules they want. Propagation then generates a DFE detection rule and a per-source hunt for each selection, running against a per-source Sigma VIEW that exposes the @@ -35,7 +37,7 @@ flowchart LR A provider fetches Sigma rules from an external feed and normalises each via pySigma into a `SigmaRuleDoc` (a mandatory UUID `id` plus `modified`/`date` change -signals, tagged with a provenance `origin`). Three kinds ship: +signals, tagged with a provenance `origin`). Three kinds are built, and each works through the API with `sigma.enabled` on. The console has no page for providers or their sync status yet. | Kind | Source | Auth | |---|---|---| @@ -58,6 +60,8 @@ edit (via `deep_merge`, where the operator's lists replace upstream's), reportin changed. The operator then SELECTS the rules to activate. Selection is its own gitcrud object. +Writing a new rule is not built. `PUT /api/v1/sigma/catalogue/{rule_id}` edits a rule already in the catalogue and answers 404 for a new id, and the console has no catalogue browser or rule editor. + ## Propagation - selection to live detections `SigmaPropagator.propagate()` walks the selection and, for each selected rule, @@ -67,6 +71,8 @@ generated rule carries a `sigma_rule_id` back-reference and a `sigma_provenance` marker (the upstream change signal + a hash of the generated WHERE) so drift and hand-edits are detected. Deselecting a rule leaves its binding STALE until pruned. +Propagation runs through the API only. There is no console flow for it, and it has not yet run against real rules and a real source. + The conversion is the [pysigma-backend-clickhouse](https://pypi.org/project/pysigma-backend-clickhouse/) backend, which owns the Sigma modifiers, wildcards and LIKE, regex and CIDR escaping. `sigma_backend_clickhouse.py` subclasses it to emit the bare condition, quote field names through the engine's column quoting and match a text-indexed column through `lower(col) LIKE`. What does not convert is recorded as `failed` in the report: - A correlation rule, which is a whole query and not a condition. @@ -84,6 +90,8 @@ the `assumeNotNull(_json).path` idiom (as text or a declared type). Identifiers are validated (`_safe_ident` / `_safe_type` / parens-balance) so a definition cannot inject into the DDL. +There are no stock ECS to Sigma or CIM to Sigma maps yet, so every source maps its own fields. + ## API + RBAC The `/api/v1/sigma` router exposes provider CRUD + sync, catalogue browse/edit/ @@ -94,6 +102,8 @@ wildcard - the SSRF/path-reach surface). See [rbac.md](../control-plane/rbac.md) ## Query cost attribution + the cost leaderboard +This section and the next are not part of Sigma, and they work with `sigma.enabled` off. + Every ClickHouse query the engine issues carries a JSON `DfeQueryTags` `log_comment` (service / feature / tenant / kind / id), stamped by the `ClickHouseClientWrapper` from a request-scoped `tags_context` (feature = the API route) or, for a hunt run, From eb5f6521f7ecd57e0c958b58177035fd88fd9caa Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 09:21:20 +1100 Subject: [PATCH 2/4] docs(hunts): say where the maths section comes from The maths section's callout says the maths is hand-written, scanned from a notebook and set in type. --- docs/data-plane/hunt-schedule-smoothing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data-plane/hunt-schedule-smoothing.md b/docs/data-plane/hunt-schedule-smoothing.md index 9048d16c..330d517c 100644 --- a/docs/data-plane/hunt-schedule-smoothing.md +++ b/docs/data-plane/hunt-schedule-smoothing.md @@ -99,7 +99,7 @@ J trending down is the loop working. J low and flat is settled. The runner's own ## Derek's maths geek section -> With apologies for Derek's maths-geek section here +> With apologies for Derek's maths-geek section here. This is Derek's hand-written maths, scanned from his notebook and set in type. ### Setup From 07122a2937549b37e5d3ea03bc21c8c51cdbe2cc Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:15:07 +1100 Subject: [PATCH 3/4] docs(hunts): document 2.2.3 smoothing beta and tests Smoothing is off by default. Turn it on with hunts.smoothing.enabled. Turning it off restores the hash offsets. It is not proven at PB/day. The lease is held for one whole cycle, so the fleet runs one cycle per window. A move can come from any bucket above the mean. Hysteresis uses the curve's variance. The blurred tie-break moves a hunt away from similar hunts. A move changes nothing a running hunt holds. Document three odd-window cases in place of two. Add cost_quantile and horizon_max_seconds to the settings table. State which settings are fixed in code. Add the two gauges from the archive to the metrics. Document three tests: - Property tests on the maths. - Tests against a real ClickHouse. - The 30-hunt soak on a 3-replica ClickHouse. Date the soak's flag-off and flag-on numbers. State what the soak does not prove. Refs #864. --- docs/data-plane/hunt-schedule-smoothing.md | 39 ++++++++++++++-------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/docs/data-plane/hunt-schedule-smoothing.md b/docs/data-plane/hunt-schedule-smoothing.md index 330d517c..47968a52 100644 --- a/docs/data-plane/hunt-schedule-smoothing.md +++ b/docs/data-plane/hunt-schedule-smoothing.md @@ -8,9 +8,9 @@ # Hunt schedule smoothing -Smoothing is held back from 2.2.2 until it has scaled testing, so no setting turns it on in this release. It comes in 2.2.3 as an opt-in beta behind `hunts.smoothing.enabled`, off by default. Tracked in https://github.com/hyperi-io/dfe-engine/issues/864. +Smoothing is a beta in 2.2.3, off by default. Turn it on with `hunts.smoothing.enabled: true` (`DFE_ENGINE_HUNTS_SMOOTHING_ENABLED=true` on the hunt runner), watch `dfe_hunt_smoothing_flatness`, and turn it off again if you don't like what you see: the schedule returns to the hash offsets. It has been tested against a 3-replica ClickHouse with 30 colliding hunts (the numbers are under "How we know it works"). It has not been proven at PB/day. Tracked in https://github.com/hyperi-io/dfe-engine/issues/864. -What 2.2.2 ships is the fixed offset: each rate hunt fires at an offset hashed from its id, within the first `hunts.phase_spread_fraction` of its interval (default 0.8, see [hunt-runner-scaling.md](hunt-runner-scaling.md#why-keda-and-a-worker-agree)). Every hunt query is also tagged with a JSON `log_comment`, so ClickHouse attributes its cost to the hunt. +Without the flag the schedule is the fixed offset: each rate hunt fires at an offset hashed from its id, within the first `hunts.phase_spread_fraction` of its interval (default 0.8, see [hunt-runner-scaling.md](hunt-runner-scaling.md#why-keda-and-a-worker-agree)). Every hunt query is tagged with a JSON `log_comment` naming the hunt and its fire, so ClickHouse attributes its cost to the hunt. Rate hunts (`schedule.mode: rate`, the default) fire on fixed intervals. Left alone, the heavy ones stack into ClickHouse CPU and memory peaks. Smoothing learns what each hunt costs and drifts WHEN it fires, never how often, so the sum flattens over time. Anchored hunts and one-off monsters are meant to spike, so the loop leaves them alone. @@ -36,10 +36,10 @@ So the loop weights both, and memory is also a hard constraint: no move may push 1. The operator's interval is untouched. Only the offset into it moves. 2. Every hunt query carries a JSON `log_comment` naming the hunt and its fire. ClickHouse records that in `system.query_log`, and the `query_log_archive` view keeps it with the real cost columns. -3. One runner replica leads. Each cycle it reads every hunt's trailing p95 cost from the archive, sums the pulses into a load curve over the hyperperiod, finds the tallest bucket, and moves ONE hunt that fires into it by a bounded step toward the deepest trough. A move is kept only if it lowers the objective by more than a hysteresis margin, and a hunt just moved sits out a cool-down. -4. The new offset is written to `hunt_schedule`, and in the beta the runner, the API's next-due and the KEDA due count all read that one table, so every reader agrees on the next fire. In 2.2.2 only the KEDA due count reads it, and the runner and the API compute the hash offset themselves. +3. One runner leads each cycle, through a lease in the schema lock table that lasts the whole cycle, so the fleet runs one cycle per window whichever runner wins. The leader reads every hunt's trailing cost from the archive, sums the pulses into a load curve over the hyperperiod, and moves ONE hunt from a bucket above the mean by a bounded step to the offset that lowers the objective most. A move is kept only if it lowers the variance of the curve by more than the hysteresis margin, and a hunt just moved sits out a cool-down. Ties between offsets break on the curve blurred 3, 7 and 15 buckets wide, which slopes toward a hole several buckets away, so a hunt stuck in a pack of alike hunts still walks out one step per cycle. +4. The new offset is written to `hunt_schedule`. The runner, the API's next-due and the KEDA due count all read that one table, so every reader agrees on the next fire. -Moving an offset changes one window's length and nothing else. A hunt moved later runs one longer window. A hunt moved earlier runs one shorter window. Coverage stays contiguous because every window starts at the last watermark. +A move writes one row in `hunt_schedule` and touches nothing a running hunt holds: not its lease, not its watermark, not an operator's run-now. Every window starts at the last watermark, so coverage stays contiguous whatever the move. The cost is one odd window: moved later, the hunt runs one extra short window, moved earlier before its fire it runs one shorter window, moved earlier after its fire one longer window. A hunt mid-run finishes under its lease, and the lease check runs before any claim, so a move never double-runs a fire. With the flag off, nothing reads the archive, nothing leads, and every offset is the hash. Turning the flag off after a run rewrites `hunt_schedule` back to the hash offsets on the runner's next start. @@ -47,7 +47,7 @@ Rescheduling fixes contention and nothing else. A query that busts the memory li ## What a hunt costs -Per hunt, per fire, over a trailing window (`hunts.smoothing.window_days`): +Per hunt, per fire, over a trailing window (`hunts.smoothing.window_days`), at the quantile `hunts.smoothing.cost_quantile` (0.95): $$ m_i = \operatorname{p95}_{\text{fires}}\bigl(\max \texttt{memory\_usage}\bigr), \qquad @@ -68,38 +68,51 @@ On ClickHouse Cloud `system.query_log` is per node, and a server upgrade can ren ## Settings -The 2.2.3 beta's settings. None of them exists in 2.2.2. - | Setting | Default | Meaning | |---|---|---| | `hunts.smoothing.enabled` | false | the master switch | | `hunts.smoothing.cycle_seconds` | 300 | how often the leader runs a cycle | | `hunts.smoothing.bucket_seconds` | 30 | Delta, the load curve's resolution | -| `hunts.smoothing.window_days` | 7 | the trailing window the p95 costs come from | +| `hunts.smoothing.window_days` | 7 | the trailing window the costs come from | +| `hunts.smoothing.cost_quantile` | 0.95 | the quantile over a hunt's fires in the window that stands for its cost | | `hunts.smoothing.max_step_fraction` | 0.1 | the most an offset moves per cycle, as a share of its interval | | `hunts.smoothing.hysteresis` | 0.01 | the least relative drop in the objective that keeps a move | | `hunts.smoothing.cooldown_cycles` | 2 | cycles a moved hunt sits out | | `hunts.smoothing.weight_memory` | 0.5 | w_m, with w_c = 1 - w_m | | `hunts.smoothing.memory_cap_bytes` | 0 | M_cap for the hard constraint, 0 reads `max_server_memory_usage` | +| `hunts.smoothing.horizon_max_seconds` | 86400 | cap on the hyperperiod H, the least common multiple of the intervals | | `smoothing.settling_period`, `autodisable.*` | - | design only, see the top of this page | -Environment form: `DFE_ENGINE_HUNTS_SMOOTHING_`. +Environment form: `DFE_ENGINE_HUNTS_SMOOTHING_`. Fixed in code, not settings: the tie-break blurs the curve 3, 7 and 15 buckets wide, and the two observed gauges look back one hour. ## Metrics -The beta's leading runner emits these each cycle, through scalo's OTel exporter ([observability-standard.md](../deployment/observability-standard.md)): +The leading runner emits these each cycle, through scalo's OTel exporter ([observability-standard.md](../deployment/observability-standard.md)): - `dfe_hunt_smoothing_enabled` (every runner, 1 when the flag is on), `dfe_hunt_smoothing_flatness` (peak/mean of P), `dfe_hunt_smoothing_objective` (J). - `dfe_hunt_cost_mem_bytes{hunt}`, `dfe_hunt_cost_duration_ms{hunt}`. - `dfe_hunt_phase_offset{hunt}`, `dfe_hunt_phase_shift_total{hunt}`. +- `dfe_hunt_observed_overlap_peak` and `dfe_hunt_observed_mem_peak_bytes`: what the archive saw over the trailing hour, the most hunt queries finishing in one bucket and the largest summed peak memory. These are the server's own numbers, not the model's, so the two can be checked against each other. + +J trending down is the loop working. J low and flat is settled. Peak over mean is not monotone, the variance is. The runner's own `dfe_hunt_backlog`, `dfe_hunt_runs_total` and `dfe_hunt_overruns_total` ship already. + +## How we know it works + +Three readings, each proving one thing. None of them needs PB/day. + +The maths first, with no ClickHouse in the room. The optimiser is handed fleets built to hurt: ten identical hunts stacked on one offset, random fleets, a fleet whose only legal moves swap a 2 for a 1. The tests check what the design claims. The total load never changes when an offset moves. Every accepted move lowers the objective and never raises it. No move leaves its bound or its step. Hysteresis holds a tiny gain, cool-down holds a hunt just moved, the memory cap holds a move that would stack memory. A hundred identical hunts move one per cycle and never herd. The stacked ten go from a peak 20 times the mean to twice the mean, the best the window allows, and then the loop stops on its own. + +Then the plumbing, on a real ClickHouse, with time driven by a clock the test owns so nothing races a real second. With the flag off the table holds exactly the hash offsets, and turning the flag off puts them back. Materialise keeps a learned offset for an unchanged hunt and drops it when the interval changes. Real hunt fires leave the archive rows the cost model reads, grouped per fire. A cycle moves one of two colliding hunts and leaves a hunt with no cost where it is. Two runners race for the lease and exactly one leads, and the lease lasts a whole cycle so a second runner ticking seconds later does not run another. + +Then real load, on a shared 3-replica ClickHouse, run on 2026-11-16: 30 hunts on 300, 900, 1800 and 3600 s intervals, 16 of them heavy GROUP BYs over 20 million rows, their ids picked so every hash offset landed in the same 30 s of its interval. The flag stayed off for an hour, then on. The gate is two numbers the leader itself emits, read from the archive and not from the model. With the flag off, the busiest 30 s bucket held 12 hunt queries and their summed peak memory reached 8.1 GB. In the first full hour with the flag on, 9 and 6.5 GB, while the model's own flatness fell from 23.7 to about 3 and the objective fell on every one of the 21 moves, each by its 10% step. Memory is the one that tracks the model. The overlap count weighs a light hunt the same as a heavy one, so light hunts that barely move the objective can keep finishing together and hold it up while memory falls. -J trending down is the loop working. J low and flat is settled. The runner's own `dfe_hunt_backlog`, `dfe_hunt_runs_total` and `dfe_hunt_overruns_total` ship in 2.2.2. +What this does not prove: the size of the benefit at PB/day, and the code 241 cliff under a real server memory limit. Those wait for a cycle at scale, and the beta flag is what lets this ship without them. One move per cycle across the fleet also means settling time grows with fleet size: 30 colliding hunts took about two hours at the 5 minute cycle, so a fleet of hundreds settles over days at the defaults. --- ## Derek's maths geek section -> With apologies for Derek's maths-geek section here. This is Derek's hand-written maths, scanned from his notebook and set in type. +> With apologies for Derek's maths-geek section here. This is Derek's hand-written maths, scanned from his notebook and set in type, and the optimiser in `hunts/smoothing_model.py` is built from it. ### Setup From f34312d36061ec0c42e0bc2f25cdce9647a3d404 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 14:53:46 +1100 Subject: [PATCH 4/4] docs(hunts): document the soak's final hours The How we know it works section documents the whole run. Across three hours with the flag on, summed peak memory per hour was 6.5, 5.0 and 3.3 GB against 8.1 GB with it off. The busiest bucket held 9, 7 and 8 queries against 12. The model's flatness fell from 23.7 to 2.3 over 34 moves. Refs #864. --- docs/data-plane/hunt-schedule-smoothing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data-plane/hunt-schedule-smoothing.md b/docs/data-plane/hunt-schedule-smoothing.md index 47968a52..4f5ee7f6 100644 --- a/docs/data-plane/hunt-schedule-smoothing.md +++ b/docs/data-plane/hunt-schedule-smoothing.md @@ -104,7 +104,7 @@ The maths first, with no ClickHouse in the room. The optimiser is handed fleets Then the plumbing, on a real ClickHouse, with time driven by a clock the test owns so nothing races a real second. With the flag off the table holds exactly the hash offsets, and turning the flag off puts them back. Materialise keeps a learned offset for an unchanged hunt and drops it when the interval changes. Real hunt fires leave the archive rows the cost model reads, grouped per fire. A cycle moves one of two colliding hunts and leaves a hunt with no cost where it is. Two runners race for the lease and exactly one leads, and the lease lasts a whole cycle so a second runner ticking seconds later does not run another. -Then real load, on a shared 3-replica ClickHouse, run on 2026-11-16: 30 hunts on 300, 900, 1800 and 3600 s intervals, 16 of them heavy GROUP BYs over 20 million rows, their ids picked so every hash offset landed in the same 30 s of its interval. The flag stayed off for an hour, then on. The gate is two numbers the leader itself emits, read from the archive and not from the model. With the flag off, the busiest 30 s bucket held 12 hunt queries and their summed peak memory reached 8.1 GB. In the first full hour with the flag on, 9 and 6.5 GB, while the model's own flatness fell from 23.7 to about 3 and the objective fell on every one of the 21 moves, each by its 10% step. Memory is the one that tracks the model. The overlap count weighs a light hunt the same as a heavy one, so light hunts that barely move the objective can keep finishing together and hold it up while memory falls. +Then real load, on a shared 3-replica ClickHouse, run on 2026-11-16: 30 hunts on 300, 900, 1800 and 3600 s intervals, 16 of them heavy GROUP BYs over 20 million rows, their ids picked so every hash offset landed in the same 30 s of its interval. The flag stayed off for an hour, then on. The gate is two numbers the leader itself emits, read from the archive and not from the model. With the flag off, the busiest 30 s bucket held 12 hunt queries and their summed peak memory reached 8.1 GB. Over the next three hours with the flag on, the summed peak memory per hour fell to 6.5, then 5.0, then 3.3 GB, and the busiest bucket to 9, 7 and 8 queries, while the model's own flatness fell from 23.7 to 2.3 over 34 moves, each by its 10% step. Memory is the one that tracks the model. The overlap count weighs a light hunt the same as a heavy one, so light hunts that barely move the objective can keep finishing together and hold it up while memory falls. What this does not prove: the size of the benefit at PB/day, and the code 241 cliff under a real server memory limit. Those wait for a cycle at scale, and the beta flag is what lets this ship without them. One move per cycle across the fleet also means settling time grows with fleet size: 30 colliding hunts took about two hours at the 5 minute cycle, so a fleet of hundreds settles over days at the defaults.