feat(anomaly-detector): hourly LLM-based journal anomaly detection

[gsanchietti]

Adds an opt-in hourly anomaly detector to ns8-loki, alongside syslog-forwarder and cloud-log-manager-forwarder. Disabled by default; requires an explicit API key.

Implements the approved design in docs/superpowers/specs/2026-07-29-loki-anomaly-detector-design.md.

What it does

A systemd timer fires a Type=oneshot service hourly. One run:

1. Takes the previous full hour as its window (wall clock, so no cursor file and no drift).
2. Queries Loki for a per-(module_id, priority) rate digest, a 7-day baseline, up to max_lines prefiltered lines (PRIORITY < 5 or category="security"), and its own last 10 findings.
3. Scrubs likely secrets from everything before it leaves the process.
4. POSTs an OpenAI-compatible chat completion demanding a JSON-schema-constrained answer.
5. Prints validated findings as JSON lines on stdout — which under systemd is the journal, so the collector ships them back into Loki. That round trip is the detector's memory: no state file, nothing extra to back up.

Skipping the LLM call on an idle window keeps quiet nodes cheap. A summary line is still written every window, so nominal hours are graphable.

Manual test

Install it on an existing machine:

runagent -m loki1
cd ../bin
curl https://raw.githubusercontent.com/NethServer/ns8-loki/refs/heads/anomaly_detector/imageroot/bin/anomaly-detector > anomaly-detector

Run using an OpenAI model:

ANOMALY_LLM_BASE_URL=https://api.openai.com/v1  ANOMALY_LLM_MODEL=gpt-4o-mini ANOMALY_LLM_API_KEY=sk-proj-xxx python3 anomaly-detector --since 2h --pretty --no-webhook

Run using a free model with OpenRouter:

ANOMALY_LLM_BASE_URL=https://openrouter.ai/api/v1 ANOMALY_LLM_MODEL=nvidia/nemotron-3-ultra-550b-a55b:free ANOMALY_LLM_API_KEY=sk-or-v1-xxx python3 anomaly-detector --since 2h --pretty --no-webhook

Configuration

api-cli run module/loki1/set-anomaly-detector --data '{
  "active": true,
  "base_url": "https://api.openai.com/v1",
  "model": "gpt-4o-mini",
  "api_key": "sk-..."
}'

base_url is a plain configured value, so OpenAI, OpenRouter, vLLM, Ollama or a self-hosted gateway all work through one code path.

Security

• The API key goes to state/secrets.env at mode 0600, never through agent.set_env — that writes state/environment, which is mirrored into the Redis hash module/<id>/environment. Verified on a live node: the Redis leak check returns nothing, and get-configuration reports only api_key_configured.
• Log lines are scrubbed inbound and findings scrubbed again outbound. IPs, hostnames, module IDs and usernames are deliberately kept — they carry the signal.
• Log text is attacker-influenced (public-internet sshd lines reach the prompt by design), so every line is flattened to one line before entering the prompt's fenced blocks. A crafted message cannot forge extra lines or close a fence.
• The detector excludes its own SYSLOG_IDENTIFIER before the priority filter, so its stderr diagnostics (journald records them at PRIORITY=3) cannot feed back into the next window.
• The README states plainly that enabling this sends log text to a third-party API. Scrubbing is defence in depth, not a guarantee.

Verification

• 160 unit tests, offline, in a container (./test-unit.sh) — new pytest harness plus a CI job.
• Verified against a live cluster: LogQL queries, SyslogIdentifier=%u/%N resolving to loki1/anomaly-detector (the module_id label the recall query depends on), the action's secret handling and disable path, and get-configuration output.
• Verified against real LLM endpoints. Worth knowing: free models are unreliable here. Two returned schema-valid JSON wrapped in a markdown fence, and one omitted a required field despite strict: true. The detector unwraps a whole-response fence, then validates exactly as strictly as before; anything that does not validate is logged truncated and exits non-zero rather than emitting a half-understood finding.

The Robot suite in tests/20__anomaly_detector.robot has not run yet — it needs CI. It uses a local stub returning a canned OpenAI-shaped completion, so there is no egress and no cost. Draft until that goes green.

Deviations from the spec

Five, each deliberate and documented in the plan: the script is import-safe with a main() so its pure functions are testable; the digest and baseline use Loki's HTTP instant-query API rather than logcli, whose metric output format is not a stable contract; scrub order puts the e-mail rule before the blob rule; the nominal path still emits a summary line; and unit tests use pytest rather than the Robot-based ns8-core script the spec cited.

Review

A whole-branch review found five substantive issues, all fixed in fbc8ab2: the e-mail scrub rule was redacting systemd unit names like agent@nethvoice2.service, stripping module names from exactly the lines that identify a crash loop; recall_findings kept the oldest findings instead of the newest; recalled titles bypassed line flattening; restore left a live API key on disk for a detector that could never run; and the Robot suite could pass without ever exercising the LLM path.