Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/data-plane/hunt-runner-scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ hunt_schedule(hunt_id, interval_seconds, phase_offset, enabled, updated)
ReplacingMergeTree(updated) ORDER BY hunt_id
```

`enabled` is a soft tombstone: a deleted hunt is written `enabled=0` so it stops counting as due, without a DELETE. The other coordination tables (`src/dfe_engine/hunts/ch_coordinator.py`) are `hunt_lease`, `hunt_watermark`, `hunt_state`, `hunt_run` and `hunt_runner_heartbeat`. `hunt_schedule` is the only one written outside the worker loop.
`enabled` is a soft tombstone: a deleted hunt is written `enabled=0` so it stops counting as due, without a DELETE. A hunt whose YAML says `enabled: false` is tombstoned the same way, since the loader keeps it out of the specs the materialiser publishes. The other coordination tables (`src/dfe_engine/hunts/ch_coordinator.py`) are `hunt_lease`, `hunt_watermark`, `hunt_state`, `hunt_run` and `hunt_runner_heartbeat`. `hunt_schedule` is the only one written outside the worker loop.

The shim's `/keda/hunt-backlog` endpoint runs `schedule.due_query(db)`, which counts hunts that are due and unclaimed. It uses the same arithmetic the worker uses (`schedule.latest_fire`):

Expand Down
9 changes: 6 additions & 3 deletions docs/data-plane/hunt-schedule-smoothing.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@

# Hunt schedule smoothing

Smoothing is a beta in 2.2.3, off by default. Turn it on with `hunts.smoothing.enabled: true` (`DFE_ENGINE_HUNTS_SMOOTHING_ENABLED=true` on the hunt runner), watch `dfe_hunt_smoothing_flatness`, and turn it off again if you don't like what you see: the schedule returns to the hash offsets. It has been tested against a 3-replica ClickHouse with 30 colliding hunts (the numbers are under "How we know it works"). It has not been proven at PB/day. Tracked in https://github.com/hyperi-io/dfe-engine/issues/864.

Check warning on line 11 in docs/data-plane/hunt-schedule-smoothing.md

View workflow job for this annotation

GitHub Actions / ci / Quality

hyperi-ci markdownlint: MD034/no-bare-urls

Bare URL used [Context: "https://github.com/hyperi-io/d..."]

Without the flag the schedule is the fixed offset: each rate hunt fires at an offset hashed from its id, within the first `hunts.phase_spread_fraction` of its interval (default 0.8, see [hunt-runner-scaling.md](hunt-runner-scaling.md#why-keda-and-a-worker-agree)). Every hunt query is tagged with a JSON `log_comment` naming the hunt and its fire, so ClickHouse attributes its cost to the hunt.

Rate hunts (`schedule.mode: rate`, the default) fire on fixed intervals. Left alone, the heavy ones stack into ClickHouse CPU and memory peaks. Smoothing learns what each hunt costs and drifts WHEN it fires, never how often, so the sum flattens over time. Anchored hunts and one-off monsters are meant to spike, so the loop leaves them alone.

Still design only, not in the 2.2.3 beta: the ClickHouse `WORKLOAD 'hunts'` backstop, promoting a settled schedule into git, and auto-disable.
Still design only, not in the 2.2.3 beta: the ClickHouse `WORKLOAD 'hunts'` backstop and promoting a settled schedule into git. Auto-disable ships in 2.2.3, outside this flag, and is described below.

## The picture

Expand Down Expand Up @@ -43,7 +43,7 @@

With the flag off, nothing reads the archive, nothing leads, and every offset is the hash. Turning the flag off after a run rewrites `hunt_schedule` back to the hash offsets on the runner's next start.

Rescheduling fixes contention and nothing else. A query that busts the memory limit on its own busts it at 3am too, and moving it changes nothing. The ladder for that is still design only: smooth, then back off the interval of a hunt that runs into itself, then auto-disable a hunt that trips code 241 K times in a window, written as a governed-ops `enabled: false` with the reason, alerted, and never re-enabled without a human.
Rescheduling fixes contention and nothing else. A query that busts the memory limit on its own busts it at 3am too, and moving it changes nothing. So the runner stops that hunt itself, smoothing flag or not: `hunts.autodisable.failures` resource-limit failures (codes 241, 158, 201) inside `hunts.autodisable.window_seconds` stop it, 3 in an hour by default. When and why land in `hunt_state` and on the hunt list, `hunt_autodisabled_total{hunt_id,reason}` counts it, and the hunt's alert destinations get the reason. Only a human brings it back, with `PUT /api/v1/hunts/{id}` naming `enabled: true` or Re-enable in the console, and an edit that leaves `enabled` alone leaves the stop alone. Backing off the interval of a hunt that runs into itself is still design only.

## What a hunt costs

Expand Down Expand Up @@ -81,7 +81,10 @@
| `hunts.smoothing.weight_memory` | 0.5 | w_m, with w_c = 1 - w_m |
| `hunts.smoothing.memory_cap_bytes` | 0 | M_cap for the hard constraint, 0 reads `max_server_memory_usage` |
| `hunts.smoothing.horizon_max_seconds` | 86400 | cap on the hyperperiod H, the least common multiple of the intervals |
| `smoothing.settling_period`, `autodisable.*` | - | design only, see the top of this page |
| `hunts.autodisable.enabled` | true | the runner stops a hunt that keeps tripping a resource limit (outside the smoothing flag) |
| `hunts.autodisable.failures` | 3 | resource-limit failures inside the window that stop it |
| `hunts.autodisable.window_seconds` | 3600 | the window those failures are counted over |
| `smoothing.settling_period` | - | design only, see the top of this page |

Environment form: `DFE_ENGINE_HUNTS_SMOOTHING_<SETTING>`. Fixed in code, not settings: the tie-break blurs the curve 3, 7 and 15 buckets wide, and the two observed gauges look back one hour.

Expand Down
74 changes: 73 additions & 1 deletion openapi/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -15020,7 +15020,7 @@
"hunts"
],
"summary": "Get Hunt",
"description": "Get full hunt configuration by file name.",
"description": "Get full hunt configuration by file name, with the runner's stop on it if any.",
"operationId": "hunts.get",
"parameters": [
{
Expand Down Expand Up @@ -41369,6 +41369,18 @@
],
"title": "Explain Queries"
},
"enabled": {
"anyOf": [
{
"type": "boolean"
},
{
"type": "null"
}
],
"title": "Enabled",
"description": "Whether the runner fires this hunt. Omitted on an update keeps the stored value. True also clears a stop the runner put on the hunt."
},
"rules": {
"items": {
"type": "string"
Expand Down Expand Up @@ -41504,6 +41516,12 @@
"title": "Name",
"description": "Hunt file name (YAML stem)"
},
"enabled": {
"type": "boolean",
"title": "Enabled",
"description": "Whether the runner fires this hunt. False keeps its YAML, state and watermark and fires nothing.",
"default": true
},
"rules": {
"items": {
"$ref": "#/components/schemas/HuntRuleEntry"
Expand All @@ -41522,6 +41540,24 @@
"type": "object",
"title": "Customer Tenant Ids",
"description": "Each customer's tenant ids, resolved when the hunt was saved: the _org_id values the runner reads. A customer missing here stops every run."
},
"auto_disabled_at": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"title": "Auto Disabled At",
"description": "When the runner stopped the hunt after repeated resource-limit failures, epoch seconds"
},
"auto_disabled_reason": {
"type": "string",
"title": "Auto Disabled Reason",
"description": "The ClickHouse error category and code that tripped the stop",
"default": ""
}
},
"type": "object",
Expand Down Expand Up @@ -41758,6 +41794,30 @@
],
"title": "Next Due",
"description": "Epoch seconds of the next scheduled fire; null = no rate schedule"
},
"enabled": {
"type": "boolean",
"title": "Enabled",
"description": "Whether the runner fires this hunt. False keeps its YAML, state and watermark and fires nothing.",
"default": true
},
"auto_disabled_at": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"title": "Auto Disabled At",
"description": "When the runner stopped the hunt after repeated resource-limit failures, epoch seconds"
},
"auto_disabled_reason": {
"type": "string",
"title": "Auto Disabled Reason",
"description": "The ClickHouse error category and code that tripped the auto-disable",
"default": ""
}
},
"type": "object",
Expand Down Expand Up @@ -41879,6 +41939,18 @@
],
"title": "Explain Queries"
},
"enabled": {
"anyOf": [
{
"type": "boolean"
},
{
"type": "null"
}
],
"title": "Enabled",
"description": "Whether the runner fires this hunt. Omitted on an update keeps the stored value. True also clears a stop the runner put on the hunt."
},
"rules": {
"items": {
"type": "string"
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ readme = "README.md"
dependencies = [
# [kafka] supplies genson to scalo.kafka.providers. Without [metrics], /metrics 404s.
"scalo[api,cli,expression,http,kafka,metrics,secrets-kubernetes]>=2.32.7,<3",
# Floor: the first release with catalogue/.
"dfe-schemas>=0.2.29,<1",
# Floor: hunt_state 1.1.0, the auto-disable columns the runner writes.
"dfe-schemas>=0.2.30,<1",
"pandas>=3.0.3",
"numpy>=2.5.4",
"clickhouse-connect>=1.10.0",
Expand Down
98 changes: 95 additions & 3 deletions src/dfe_engine/api/v1/hunts.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,20 @@
from dfe_engine.api.write_turn import WRITE_TURN
from dfe_engine.auth.audit import audit_resource_change
from dfe_engine.auth.rbac_scopes import scopes_dict
from dfe_engine.gitcrud.routing import WriteOutcome
from dfe_engine.hunts.alert_hunt_link import delete_destinations_owned_by_hunt
from dfe_engine.hunts.customers import CUSTOMER_TENANT_IDS, hunt_customers
from dfe_engine.hunts.hunt_config_registry import (
HuntConfigNotFoundError,
default_display_name,
resolve_display_name,
)
from dfe_engine.hunts.run_status import RunStatus, live_runner_count, read_run_status
from dfe_engine.hunts.run_status import (
RunStatus,
clear_auto_disable,
live_runner_count,
read_run_status,
)
from dfe_engine.hunts.schedule import next_due, read_schedule
from dfe_engine.hunts.spec_loader import interval_seconds_for
from dfe_engine.hunts.validator import HuntValidator
Expand Down Expand Up @@ -185,6 +191,23 @@ class HuntSummary(BaseModel):
default=None,
description="Epoch seconds of the next scheduled fire; null = no rate schedule",
)
enabled: bool = Field(
default=True,
description=(
"Whether the runner fires this hunt. False keeps its YAML, state and watermark "
"and fires nothing."
),
)
auto_disabled_at: int | None = Field(
default=None,
description=(
"When the runner stopped the hunt after repeated resource-limit failures, epoch seconds"
),
)
auto_disabled_reason: str = Field(
default="",
description="The ClickHouse error category and code that tripped the auto-disable",
)


class HuntRuleEntry(BaseModel):
Expand Down Expand Up @@ -226,6 +249,15 @@ class _HuntConfigFields(BaseModel):
class HuntWriteRequest(_HuntConfigFields):
"""Hunt scheduler payload for create/update (without hunt file ``name``)."""

# The read model carries its own enabled flag: here None keeps the stored flag on
# an update, so an edit of the cron never re-enables a hunt an operator stopped.
enabled: bool | None = Field(
default=None,
description=(
"Whether the runner fires this hunt. Omitted on an update keeps the stored "
"value. True also clears a stop the runner put on the hunt."
),
)
rules: list[str] = Field(
min_length=1,
description="Rule names, each the ``{name}.yaml`` rule file the hunt runner reads",
Expand Down Expand Up @@ -268,6 +300,13 @@ class HuntDetailResponse(_HuntConfigFields):

name: str = Field(description="Hunt file name (YAML stem)")
display_name: str = Field(description="Resolved human-readable label") # ty: ignore[invalid-mutable-override] # the response model narrows the field, never writes it
enabled: bool = Field(
default=True,
description=(
"Whether the runner fires this hunt. False keeps its YAML, state and watermark "
"and fires nothing."
),
)
rules: list[HuntRuleEntry] = Field(min_length=1)
# A stored hunt naming none is shown, and runs nothing.
customers: list[str] = Field(
Expand All @@ -280,6 +319,15 @@ class HuntDetailResponse(_HuntConfigFields):
"values the runner reads. A customer missing here stops every run."
),
)
# The runner's stop rides on GET alone: a save answers with what it stored, and a
# save naming enabled true has just cleared the stop.
auto_disabled_at: int | None = Field(
default=None,
description="When the runner stopped the hunt after repeated resource-limit failures, epoch seconds",
)
auto_disabled_reason: str = Field(
default="", description="The ClickHouse error category and code that tripped the stop"
)

@classmethod
def from_stored_config(cls, name: str, config: dict[str, Any]) -> HuntDetailResponse:
Expand Down Expand Up @@ -383,6 +431,39 @@ def _run_status_for(hunt_names: list[str], now: int) -> dict[str, RunStatus]:
return {}


def _clear_auto_disable(name: str, *, enabled: bool | None, outcome: WriteOutcome | None) -> None:
"""Lift the runner's stop on a hunt when a direct save names ``enabled: true``.

An edit that omits ``enabled`` leaves the stop where it is, and so does a save
routed to a review branch, since nothing has changed on the branch the runner
reads. The YAML is saved by now, so a failure is told to the caller as 503
rather than answering 200 over a hunt that is still stopped.

Raises:
ProblemError: 503 ``auto_disable_not_cleared`` when ClickHouse did not take it.
"""
if enabled is not True or (outcome is not None and outcome.review_required):
return
try:
client, db = _data_database_client()
run = read_run_status(client, db, [name], int(time.time())).get(name)
if run is None or run.auto_disabled_at is None:
return
clear_auto_disable(client, db, name)
except Exception as exc:
raise ProblemError(
503,
f"Hunt '{name}' was saved, but the runner's stop on it could not be cleared: "
f"{exc}. Repeat the request once ClickHouse answers.",
code="auto_disable_not_cleared",
) from exc


def _enabled_flag(value: Any) -> bool:
"""A stored ``enabled`` as the flag the runner reads: absent is on, anything but True is off."""
return value is None or value is True


def _learned_offsets(enabled: bool) -> dict[str, tuple[int, int]]:
"""The offsets ``hunt_schedule`` holds, read once per listing when smoothing is on.

Expand Down Expand Up @@ -441,6 +522,9 @@ def _hunt_row_to_summary(
too_aggressive=run.too_aggressive,
run_requested=run.run_requested,
next_due=_next_due(row, now, fraction, learned),
enabled=_enabled_flag(row.get("enabled")),
auto_disabled_at=run.auto_disabled_at,
auto_disabled_reason=run.auto_disabled_reason,
)


Expand Down Expand Up @@ -572,12 +656,17 @@ def get_hunt(
user: CurrentUser,
registry: HuntConfigReg,
) -> HuntDetailResponse:
"""Get full hunt configuration by file name."""
"""Get full hunt configuration by file name, with the runner's stop on it if any."""
try:
config = registry.get(name)
except HuntConfigNotFoundError:
raise ProblemError(404, f"Hunt '{name}' not found", code="not_found") from None
return HuntDetailResponse.from_stored_config(name, config)
detail = HuntDetailResponse.from_stored_config(name, config)
run = _run_status_for([name], int(time.time())).get(name)
if run is not None:
detail.auto_disabled_at = run.auto_disabled_at
detail.auto_disabled_reason = run.auto_disabled_reason
return detail


@router.put(
Expand Down Expand Up @@ -608,9 +697,12 @@ def update_hunt(

_refuse_new_rule_names(body.rules, stored)
config = body.to_config_dict(hunt_name=name)
if body.enabled is None and "enabled" in stored:
config["enabled"] = stored["enabled"]
_validate_hunt_config(config, settings, rules)
_resolve_customers(request, config)
outcome = registry.save(name, config, created_by=user.user_id)
_clear_auto_disable(name, enabled=body.enabled, outcome=outcome)
apply_review_headers(response, outcome)
audit_resource_change(user.user_id, "hunt", name, "updated", review_audit_detail(outcome))
return HuntDetailResponse.from_stored_config(name, config)
Expand Down
1 change: 1 addition & 0 deletions src/dfe_engine/clickhouse/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ class _Meta:
164: _Meta(ErrorCategory.ACCESS, user_safe=True), # READONLY
192: _Meta(ErrorCategory.ACCESS, user_safe=True), # UNKNOWN_USER
193: _Meta(ErrorCategory.ACCESS, user_safe=True), # WRONG_PASSWORD
201: _Meta(ErrorCategory.RESOURCE_LIMIT, user_safe=True), # QUOTA_EXCEEDED
202: _Meta(ErrorCategory.RATE_LIMITED, user_safe=True), # TOO_MANY_SIMULTANEOUS_QUERIES
209: _Meta(ErrorCategory.CONNECTION), # SOCKET_TIMEOUT
210: _Meta(ErrorCategory.CONNECTION), # NETWORK_ERROR
Expand Down
4 changes: 4 additions & 0 deletions src/dfe_engine/hunts/alert.py
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,10 @@ def _should_fire(self, result_count: int, results: list[dict[str, Any]] | None)

return False

def send(self, title: str, body: str) -> bool:
"""Send one notification to every destination, with no trigger evaluated."""
return self._send(title, body)

def _send(self, title: str, body: str) -> bool:
"""Send notification via Apprise. Returns True on success."""
try:
Expand Down
Loading
Loading