Skip to content

fix(hunts): enabled flag, auto-disable on limits - #1191

Merged
catinspace-au merged 7 commits into
mainfrom
fix/hunt-disable
Nov 16, 2026
Merged

catinspace-au merged 7 commits into
mainfrom
fix/hunt-disable

Conversation

@catinspace-au

@catinspace-au catinspace-au commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1179. Follows the smoothing beta in #1185.

Hunt YAML has enabled, default true. A disabled hunt stays out of the runner's specs and fires nothing. KEDA stops counting it. Its YAML, state and watermark stay.

The validator and loader take only a YAML bool. A quoted "false" would otherwise read as on.

Auto-disable is on by default. The runner stops a hunt after hunts.autodisable.failures ClickHouse resource limit errors inside hunts.autodisable.window_seconds. The errors are 241 memory, 158 rows, 201 quota.

The runner records when and why in hunt_state. It increments hunt_autodisabled_total. It alerts through the hunt's destinations with the clear command. It skips the hunt every tick after.

A PUT with enabled: true clears the auto-disable. A cron edit leaves it disabled.

The API shows enabled, auto_disabled_at and auto_disabled_reason on the hunt list and detail. The change needs dfe-schemas 0.2.30 or later. That release has hunt_state 1.1.0.

argMax skips a NULL argument. A cleared stop read as still stopped until the merge collapsed the rows. Both reads take the latest row as a tuple.

The two data-plane docs pages wait for #1177 to merge, since both live on that branch. The smoothing page documents shipped auto-disable behaviour. hunt-runner-scaling documents that a disabled hunt is tombstoned in hunt_schedule.

On desktop-derek, 6653 unit tests passed across test_hunts, test_api, test_settings and test_clickhouse. hyperi-ci check --quick passed. The hunt integration files had 29 passing tests across auto-disable, run visibility, runner e2e, pipeline, reload and smoothing. The full suite was not run.

The runner stops hunts that keep hitting a ClickHouse resource limit. The API shows the auto-disable and lets an operator clear it.

- hunts.autodisable settings: enabled (on), failures (3) and window_seconds (3600), each settable through DFE_ENGINE_HUNTS_AUTODISABLE_* environment variables
- hunt_autodisabled_total counter, labelled by hunt_id and reason
- RunStatus reads auto_disabled_at and auto_disabled_reason from hunt_state.
- clear_auto_disable writes a row to replace the disabled row.
- Hunt create, update, list and detail have enabled.
- The list adds auto_disabled_at and auto_disabled_reason.
- Saving a hunt with enabled true tries to clear the auto-disable. The save succeeds if ClickHouse is unreachable.
- The hunt list reads enabled from the stored YAML
Hunt YAML has `enabled`, default true. Disabled hunts stay out of the runner's specs and fire nothing. KEDA stops counting them. Their YAML, state and watermark stay.

The validator and loader take only YAML booleans. A quoted "false" would otherwise read as on.

The runner stops a hunt that trips a ClickHouse resource limit (241 memory, 158 rows, 201 quota) `hunts.autodisable.failures` times inside `hunts.autodisable.window_seconds`. It records when and why in hunt_state and increments `hunt_autodisabled_total`. It alerts through the hunt's destinations with the clear command. It skips the hunt every tick after.

A PUT with `enabled: true` clears the auto-disable. A cron edit leaves it disabled.

The API shows `enabled`, `auto_disabled_at` and `auto_disabled_reason` on the hunt list and detail. hunt_state 1.1.0 (dfe-schemas #128) has the two columns.
argMax skips a NULL argument. `argMax(auto_disabled_at, updated)` kept an older stop time after the API cleared it with NULL, until the merge collapsed the rows. The runner's auto-disabled read and the run-status join take both columns from the latest row as a tuple. The tuple is never NULL.

The minimum dfe-schemas version is 0.2.30, the release with hunt_state 1.1.0. The auto-disable integration test runs against 0.2.30:
- Two resource-limit failures stop the hunt.
- The stopped tick fires nothing.
- Clearing the stop through the API restarts the hunt.
@catinspace-au
catinspace-au marked this pull request as ready for review November 16, 2026 05:22
An update that omits `enabled` keeps the stored flag. The write model's default of true wrote `enabled: true` on every PUT, so a cron edit re-enabled a hunt an operator had stopped. The field is optional on writes. A new hunt is enabled.

The runner stores the stop in a separate hunt_state row, keyed `<hunt_id>#stopped`. The overrun counter's read-then-write cannot overwrite the separate stop row. The clear writes the marker without reading it.

Saving `enabled: true` clears the stop only if the write succeeds on the branch the runner reads. A failed clear returns 503 `auto_disable_not_cleared` instead of 200. The YAML is saved, but the hunt is still stopped.

The smoothing page documents the shipped auto-disable settings and counter. The scaling page says a YAML `enabled: false` hunt is tombstoned like a deleted one.
Both models declare `enabled`. The detail defaults to true. A write defaults to None so an omitted flag keeps the stored value.

Neither model overrides a parent field, so no type-check suppression is needed. CI and local ty results match.
dfe-schemas 0.2.30 limits LowCardinality to String, FixedString and UUID. ClickHouse rejects LowCardinality over a number without allow_suspicious_low_cardinality_types (code 455). DFE never sets it.

Type registry tests check rejection of a number, float, date or address with `lowcardinality`. The override case wraps a FixedString.
@catinspace-au
catinspace-au merged commit b342190 into main Nov 16, 2026
19 checks passed
@catinspace-au
catinspace-au deleted the fix/hunt-disable branch November 16, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-disable a hunt that keeps tripping the ClickHouse memory limit or a quota

1 participant