Repository navigation
fix(hunts): enabled flag, auto-disable on limits - #1191
Merged
Merged
Conversation
The runner stops hunts that keep hitting a ClickHouse resource limit. The API shows the auto-disable and lets an operator clear it. - hunts.autodisable settings: enabled (on), failures (3) and window_seconds (3600), each settable through DFE_ENGINE_HUNTS_AUTODISABLE_* environment variables - hunt_autodisabled_total counter, labelled by hunt_id and reason - RunStatus reads auto_disabled_at and auto_disabled_reason from hunt_state. - clear_auto_disable writes a row to replace the disabled row. - Hunt create, update, list and detail have enabled. - The list adds auto_disabled_at and auto_disabled_reason. - Saving a hunt with enabled true tries to clear the auto-disable. The save succeeds if ClickHouse is unreachable. - The hunt list reads enabled from the stored YAML
Hunt YAML has `enabled`, default true. Disabled hunts stay out of the runner's specs and fire nothing. KEDA stops counting them. Their YAML, state and watermark stay. The validator and loader take only YAML booleans. A quoted "false" would otherwise read as on. The runner stops a hunt that trips a ClickHouse resource limit (241 memory, 158 rows, 201 quota) `hunts.autodisable.failures` times inside `hunts.autodisable.window_seconds`. It records when and why in hunt_state and increments `hunt_autodisabled_total`. It alerts through the hunt's destinations with the clear command. It skips the hunt every tick after. A PUT with `enabled: true` clears the auto-disable. A cron edit leaves it disabled. The API shows `enabled`, `auto_disabled_at` and `auto_disabled_reason` on the hunt list and detail. hunt_state 1.1.0 (dfe-schemas #128) has the two columns.
argMax skips a NULL argument. `argMax(auto_disabled_at, updated)` kept an older stop time after the API cleared it with NULL, until the merge collapsed the rows. The runner's auto-disabled read and the run-status join take both columns from the latest row as a tuple. The tuple is never NULL. The minimum dfe-schemas version is 0.2.30, the release with hunt_state 1.1.0. The auto-disable integration test runs against 0.2.30: - Two resource-limit failures stop the hunt. - The stopped tick fires nothing. - Clearing the stop through the API restarts the hunt.
catinspace-au
marked this pull request as ready for review
November 16, 2026 05:22
An update that omits `enabled` keeps the stored flag. The write model's default of true wrote `enabled: true` on every PUT, so a cron edit re-enabled a hunt an operator had stopped. The field is optional on writes. A new hunt is enabled. The runner stores the stop in a separate hunt_state row, keyed `<hunt_id>#stopped`. The overrun counter's read-then-write cannot overwrite the separate stop row. The clear writes the marker without reading it. Saving `enabled: true` clears the stop only if the write succeeds on the branch the runner reads. A failed clear returns 503 `auto_disable_not_cleared` instead of 200. The YAML is saved, but the hunt is still stopped. The smoothing page documents the shipped auto-disable settings and counter. The scaling page says a YAML `enabled: false` hunt is tombstoned like a deleted one.
Both models declare `enabled`. The detail defaults to true. A write defaults to None so an omitted flag keeps the stored value. Neither model overrides a parent field, so no type-check suppression is needed. CI and local ty results match.
dfe-schemas 0.2.30 limits LowCardinality to String, FixedString and UUID. ClickHouse rejects LowCardinality over a number without allow_suspicious_low_cardinality_types (code 455). DFE never sets it. Type registry tests check rejection of a number, float, date or address with `lowcardinality`. The override case wraps a FixedString.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1179. Follows the smoothing beta in #1185.
Hunt YAML has
enabled, default true. A disabled hunt stays out of the runner's specs and fires nothing. KEDA stops counting it. Its YAML, state and watermark stay.The validator and loader take only a YAML bool. A quoted "false" would otherwise read as on.
Auto-disable is on by default. The runner stops a hunt after
hunts.autodisable.failuresClickHouse resource limit errors insidehunts.autodisable.window_seconds. The errors are 241 memory, 158 rows, 201 quota.The runner records when and why in hunt_state. It increments
hunt_autodisabled_total. It alerts through the hunt's destinations with the clear command. It skips the hunt every tick after.A PUT with
enabled: trueclears the auto-disable. A cron edit leaves it disabled.The API shows
enabled,auto_disabled_atandauto_disabled_reasonon the hunt list and detail. The change needs dfe-schemas 0.2.30 or later. That release has hunt_state 1.1.0.argMax skips a NULL argument. A cleared stop read as still stopped until the merge collapsed the rows. Both reads take the latest row as a tuple.
The two data-plane docs pages wait for #1177 to merge, since both live on that branch. The smoothing page documents shipped auto-disable behaviour. hunt-runner-scaling documents that a disabled hunt is tombstoned in hunt_schedule.
On desktop-derek, 6653 unit tests passed across test_hunts, test_api, test_settings and test_clickhouse. hyperi-ci check --quick passed. The hunt integration files had 29 passing tests across auto-disable, run visibility, runner e2e, pipeline, reload and smoothing. The full suite was not run.