This document tracks how a signal flows through the system and what each function does and why. Read it alongside:
- architectural.md — why the system is shaped this way (ADRs).
- docs/superpowers/specs/2026-08-13-intelliops-coe-design.md — the full spec.
Status (updated 2026-09-22). The six-service closed loop is built and running end-to-end, plus three things that came after the original design: a read-model service (the CQRS read side the dashboard reads from), a React operator console, and a live, repeatably-runnable demo stack on docker-compose. What ran only as a "target design" in the first draft now runs live — including remediation itself, which can now drive a real Kubernetes cluster behind an opt-in switch (dry-run stays the production-safe default). Kafka bus binding and whole-stack Helm deploy are also shipped. See §8 Current status & what's next for exactly what's real, what's still simulated, and what's next.
A sample production system now feeds real faults into this pipeline. Meridian (
services/meridian/) — a four-service financial/audit platform with its own portal UI — runs alongside IntelliOps, scraped by the same Prometheus, ingested by the same ingestion service (query broadened additively), and diagnosed by the same unmodified RCA rules. Each service emits a USE+RED metric set with 8 typed fault scenarios, each moving a realistic metric cluster. See docs/MERIDIAN.md and ADR-020.The remediation path gained a rehearsal, a wider-but-still-safe vocabulary, and bounded selection intelligence — all config-switched off by default. Before approval a fix is rehearsed on an isolated namespace clone (ADR-023); the typed action
Literalwidened 4→7 Deployment-scoped verbs behind a destructive-shape denylist (ADR-024); the AI can draft a runbook for a gap that a human approves (ADR-025); and selection is rules-first with an embedding-similarity fallback over vetted playbooks (ADR-026) — retrieval, not an LLM choosing the fix. See §5.3–5.5 for where each sits in the flow.
A single production incident, from raw telemetry to a closed feedback loop:
Prometheus / Loki / OpenTelemetry
│ (1) scrape / receive
▼
┌────────────────────┐
│ ingestion-service │ normalize → TelemetryEvent, dedup by fingerprint
└────────────────────┘ ── publish ▶ topic: telemetry.raw
│
▼
┌────────────────────┐
│ correlation-service│ (2) detect_anomaly per event
│ │ (3) correlate anomalies → one Situation
└────────────────────┘ ── publish ▶ topic: situations.detected
│
▼
┌────────────────────┐
│ rca-service │ (4) enrich with deploy/config/topology context
│ │ (5) rank root-cause hypotheses + attach runbook
└────────────────────┘ ── publish ▶ topic: situations.diagnosed
│
▼
┌────────────────────┐ ┌───────────────────────┐
│ action-service │ (6) ──▶ │ governance-service │ RBAC check + HITL approval
│ select_playbook │ ◀── ✔/�’ │ (SYNCHRONOUS gate) │ + write audit record
│ execute → verify │ └───────────────────────┘
│ rollback if unhealthy │ (audit events from ALL
└────────────────────┘ ── publish ▶ remediation.outcomes │ services flow here async)
│
▼
┌────────────────────┐
│ feedback-service │ (7) label outcome → training store
│ │ compute MTTR/MTTD metrics
└────────────────────┘ ── training data ▶ correlation-service.retrain() ⟲ LOOP CLOSED
── ALSO consuming the situation/outcome stream (the read side): ──
┌────────────────────┐
│ read-service │ projects situations.detected/diagnosed + remediation.outcomes
│ (CQRS read model) │ + situations.suppressed into the exact shapes the UI needs
└────────────────────┘ ── serves ▶ GET /situations · /outcomes · /metrics
│ ── pushes ▶ GET /stream (SSE: a "changed" nudge on every mutation)
▼
┌────────────────────┐
│ React console │ Overview (live KPIs) · Incidents (HITL approve/reject)
│ (operator UI) │ · Pipeline (live incident-journey lanes) · Governance (gates,
└────────────────────┘ audit, playbook registry) · Audit (filterable audit explorer)
The single synchronous step is action-service → governance-service (step 6). Everything
else is asynchronous over the bus. The gate fails closed: no governance approval → no
action. See ADR-003.
The read side (CQRS). The six services above are the write side — they consume, decide,
and emit events. The read-service (ADR-009)
is a separate consumer that projects those events into an in-memory read model and serves the
dashboard over plain GET endpoints. It holds no source-of-truth state — the Redis event
streams are the record, and the projection rebuilds from them on restart. The read-model also
pushes: GET /stream (Server-Sent Events) emits a generic {"type": "changed"} nudge on
every projection mutation, so the console re-fetches within about a second of a real change
instead of waiting out a poll interval (ADR-018).
| Topic | Produced by | Consumed by | Payload | Meaning |
|---|---|---|---|---|
telemetry.raw |
ingestion | correlation | TelemetryEvent |
a normalized signal |
situations.detected |
correlation | rca, read | Situation (detected) |
an alert storm collapsed into one incident |
situations.diagnosed |
rca | action, read | DiagnosedSituation (situation + hypotheses) |
incident with likely cause + suggested fix |
situations.suppressed |
correlation | read | Situation (suppressed) |
a signature the system has reliably fixed. Under the default suppression_mode=quiet it is ALSO emitted on situations.detected with handling="quiet" and remediated without paging a human when the playbook's real track record qualifies; this topic is the record of it (ADR-034) |
remediation.outcomes |
action | feedback, read | RemediationOutcome |
did the fix work / roll back |
audit.events |
all services | governance | AuditRecord |
append-only trail (fire-and-forget) |
Note.
situations.diagnosedcarries aDiagnosedSituation(theSituationplus rankedRootCauseHypothesislist and the suggested runbook id) — additive over the frozenSituationcontract. Governance also handles approvals over REST (/approvals), not the bus — approvals are request/response, not a stream (see §5.5).
These are the nouns every service passes around. Defined once in common/ so services can't
drift (ADR-006).
| Contract | Key fields | Why it exists |
|---|---|---|
TelemetryEvent |
source, kind, name, value/payload, labels, ts, fingerprint |
Canonical shape so every downstream service is source-agnostic. |
Situation |
id, status, member_events[], severity, first_seen, last_seen, signature |
The universal incident currency; signature recognizes recurrences (ADR-004). |
RootCauseHypothesis |
situation_id, description, confidence, evidence[], suggested_runbook_id |
Makes RCA output rankable and actionable, not just prose. |
DiagnosedSituation |
situation, hypotheses[], suggested_runbook_id |
The situations.diagnosed payload: a diagnosed Situation plus ranked causes. Additive — never mutates the frozen Situation. |
RemediationStep |
action, replicas, note + cpu_limit, mem_limit, container, revision, probe, … |
The typed unit of a fix. action is a closed Literal of 7 Deployment-scoped verbs — the core safety property: an unlisted action can't validate (ADR-024). |
Playbook |
id, name, match_rule, steps[], hitl_mode, reversible, rollback_steps[], symptoms |
A remediation with its safety scope and undo baked in (ADR-007). symptoms (added later) is the curated text the semantic selector matches against (ADR-026). |
PreflightResult |
passed, detail, mode, sandbox_namespace |
The sandbox rehearsal verdict (ADR-023); rides on ApprovalRequest + every RemediationOutcome. |
ProposedPlaybook |
id, playbook, status, proposed_by, rationale, source_situation_id, decided_by, ts |
An AI-drafted runbook awaiting human approval (ADR-025); the inner playbook validates via the closed action Literal. |
ApprovalRequest |
id, situation_id, playbook_id, requested_by, status, decided_by, preflight |
The record a human approves/rejects at the HITL gate; preflight (added later) shows the human the rehearsal verdict. |
RemediationOutcome |
situation_id, playbook_id, result, health_after, ts, hitl_mode, steps, mode, preflight |
The feedback signal that closes the loop. hitl_mode/steps/mode/preflight (added later) let the read model report auto-vs-HITL, the executed steps, dry-run-vs-k8s, and the sandbox verdict truthfully. |
AuditRecord |
actor, action, resource, decision, ts, correlation_id |
Immutable compliance trail; correlation_id threads one incident end to end. |
The swap points that keep the system platform-agnostic and testable (ADR-005).
| Interface | Methods | Implementations that exist today | Deferred / planned |
|---|---|---|---|
TelemetrySource |
poll(), subscribe() |
FileTelemetrySource (JSONL, tests), PrometheusSource (real PromQL over the Prometheus HTTP API, defensive: never raises on a bad response) |
Loki, OTel sources |
Correlator |
detect(), correlate(), retrain() |
RiverCorrelator (online z-score, default), RobustCorrelator (median/MAD + seasonal per-hour baseline), TrainedCorrelator (composes RobustCorrelator + a persisted scikit-learn IsolationForest, fitted via POST /retrain) — selected by CORRELATOR_KIND (ADR-019) |
— |
ExplanationProvider |
explain() |
TemplateExplanationProvider (deterministic, no network — default), OpenAICompatibleExplanationProvider (real endpoint when llm_explanation_endpoint is set; falls back to the template on any error) |
— |
RunbookSelector |
select() |
NullRunbookSelector (no semantic fallback — default, keyword-rules-only), EmbeddingRunbookSelector (local sentence-transformers, cosine similarity over playbook symptoms; ranks only registered playbooks; fail-safe → None; RUNBOOK_SELECTOR_MODE=embedding, ml extra, lazy-imported) (ADR-026) |
— |
RunbookAuthor |
draft() |
NullRunbookAuthor (returns None — default), OpenAICompatibleRunbookAuthor (drafts a typed Playbook for a gap via an OpenAI-chat endpoint; fail-to-nothing — any failure or an unsafe/invalid draft → None; RUNBOOK_AUTHOR_MODE=openai) (ADR-025) |
— |
Remediator |
execute(), rollback() |
DryRunRemediator (logs steps, touches nothing — the safe default), RecordingRemediator (tests), KubernetesRemediator (real AppsV1Api calls — restart via a restartedAt annotation patch, scale via patch_namespaced_deployment_scale, rollback via a rollout-restart annotation; no shell, no string parsing; never deletes; any API error → False, never raises. Behind REMEDIATOR_MODE=k8s, targeting a local kind cluster — see deploy/k8s/README.md) |
— |
HealthChecker |
check() |
AlwaysHealthyChecker (pairs with dry-run), FixedHealthChecker (tests), real KubernetesHealthChecker (two signals — pod readiness from deployment status, and metric recovery re-queried from Prometheus — polled to a timeout; fails closed. Behind HEALTH_CHECK_MODE=k8s) |
— |
Sandbox |
rehearse() |
NullSandbox (passes through — default, SANDBOX_MODE=off), NamespaceCloneSandbox (clones the target Deployment into a throwaway namespace, applies the same plan, watches the clone recover, tears it down; ranks the fix's effect before approval; fail-safe → PreflightResult(passed=False), never raises; SANDBOX_MODE=k8s) (ADR-023) |
— |
GovernanceGate (action→governance) |
check_rbac(), request_approval(), await_decision(), write_audit() |
InProcessGovernanceGate (shared dict, single-process/tests), HttpGovernanceGate (REST — works across containers; fail-closed on any error) |
— |
AuditSink |
write(), records() |
FileAuditSink (JSONL), InMemoryAuditSink (tests), PostgresAuditSink (hybrid schema — indexed columns + a JSONB payload that is the source of truth; errors propagate, a lost audit write is a compliance failure. Behind STORE_BACKEND=postgres — see docs/PERSISTENCE.md) |
— |
PlaybookStore |
register(), get(), list() |
InMemoryPlaybookStore (tests), FilePlaybookStore (YAML dir), PostgresPlaybookStore (upsert via ON CONFLICT; same STORE_BACKEND=postgres switch) |
— |
TrainingStore |
append(), read_all() |
InMemoryTrainingStore (tests), FileTrainingStore (JSONL), PostgresTrainingStore (same switch; errors propagate) |
— |
ApprovalStore |
create(), get(), decide(), list_pending() |
InMemoryApprovalStore (file/tests), PostgresApprovalStore (durable pending HITL approvals; errors propagate like the audit sink — same STORE_BACKEND=postgres switch; ADR-015) |
— |
BaselineStore |
save(), load_all() |
InMemoryBaselineStore (tests), PostgresBaselineStore (durable z-score baseline; snapshot/reload is best-effort, logged and fail-safe — never crashes the flusher or boot; None in file mode; ADR-015) |
— |
BusClient |
publish(), consume() |
RedisBus (Redis Streams, consumer groups) |
Kafka (prod) |
Tests bind fakes (RecordingRemediator, FixedHealthChecker, InMemory*) to exercise a
service in isolation. Selection is config-driven — env switches pick the live vs test-safe
binding (TELEMETRY_MODE=file|prometheus, GOVERNANCE_MODE=in_process|http,
STORE_BACKEND=file|postgres), so the default build stays test-safe and the compose stack turns
the live bindings on.
For each function: what it does · why we use it · what it depends on.
| Function | What it does | Why | Depends on |
|---|---|---|---|
run_poll_loop() |
On an interval, pulls from each configured TelemetrySource and hands raw signals to normalize. |
The service's heartbeat; keeps telemetry flowing without downstream services knowing the source. | TelemetrySource, BusClient |
normalize(raw) → TelemetryEvent |
Converts a vendor-specific signal (Prom sample, Loki line, OTel span) into the canonical TelemetryEvent. |
So correlation and everything after it are source-agnostic — the whole point of the data layer. | contracts.TelemetryEvent |
compute_fingerprint(event) → str |
Stable hash of the event's identity (name + labels). | Kills duplicate alerts at the door — the first, cheapest cut of the 80–95% noise-reduction goal. | — |
| Function | What it does | Why | Depends on |
|---|---|---|---|
detect_anomaly(event) → score |
Scores an event against the learned baseline; flags the abnormal ones. | The noise-reduction engine — separates signal from the 85–95% false positives. | Correlator |
correlate(anomalies) → Situation |
Clusters related anomalies (by time / topology / labels) into one Situation. |
Turns an alert storm into a single actionable incident — the core value proposition. | Correlator, contracts.Situation |
load_model() |
Loads the current anomaly/correlation model at startup. | Lets the model evolve without code changes. | model store |
retrain(training_data) |
Re-fits the model from labeled outcomes produced by feedback-service. |
This is where the closed loop lands — accuracy compounds over time. | training store, Correlator |
The detector itself is pluggable, config-selected by CORRELATOR_KIND
(ADR-019),
all three subclassing a shared BaseCorrelator:
river(default, unchanged) —RiverCorrelator: per-metric online z-score against a running mean/variance, warm-up gated.robust—RobustCorrelator: median/MAD ("robust z-score") instead of mean/variance, so one early spike no longer desensitizes the baseline, plus a seasonal baseline: a separate window per(metric, hour-of-day)bucket so a daily traffic pattern doesn't read as a global outlier. This is the detectordocs/BENCHMARKS.mdshows cutting false positives ~7x on a seasonal scenario.trained—TrainedCorrelator: composesrobustfor the live online score and blends in a scikit-learnIsolationForest's anomaly margin (max()of the two, so the model can only add sensitivity). The model is not fitted automatically —POST /retrainon correlation-service is the real trigger (boot'sretrain()call only re-aggregates reliability; the feature buffer is empty at boot, so nothing would be fit from). A fit's result is persisted as a joblib blob to amodel_artifactstable via aModelStoreand reloaded on the next boot, so eachPOST /retrainimproves on the last one and a restart keeps the trained model. A coldtrainedcorrelator (no fit yet) scores identically torobust. sklearn/joblib are imported lazily — ariver/robustdeployment never pays the import cost.
The anomaly decision itself is policy-aware. detect_anomaly still produces the
score above, but whether that score (or the raw event value) counts as an anomaly is
now decided by a DetectionPolicy shared via BaseCorrelator: kind-based absolute
thresholds for ratio/saturation/latency metrics when DETECTION_POLICY=on, else
the unmodified z-score (default off, byte-identical). See
ADR-027 for the detail.
| Function | What it does | Why | Depends on |
|---|---|---|---|
enrich(situation) → context |
Attaches recent deploys, config/change data, and service topology to the situation. | Context is what makes a root-cause suggestion credible instead of a guess. | deploy/config/topology sources |
rank_hypotheses(situation, context, reliability_provider=None, store=None, selector=None) → [RootCauseHypothesis] |
Two-layer diagnosis: keyword rules PROPOSE a candidate hypothesis per metric family (deploy→rollback-deploy; memory/db_pool/log-error→restart-pod; saturation/latency/queue_depth/request_rate→scale-service) from the closed 3-runbook catalog, each with a fallback confidence. When store+selector are supplied and RUNBOOK_SELECTOR_MODE=embedding, the selector's cosine fit of the incident's symptoms against that candidate runbook's symptoms text COMPUTES the hypothesis's confidence in place of the fallback (confidence_source="embedding"); off, or on any embedding error, the fallback constant stands (confidence_source="rule") — never raises. A reliability_provider, if passed, then applies its bounded boost on top. |
Gives responders a ranked starting point, not a wall of data; lets the AI's fit-computed confidence — not a hand-tuned constant — decide how strongly a candidate is recommended, while the rules alone still decide which runbook is even in play (ADR-028). | contracts.RootCauseHypothesis, training_store, RunbookSelector |
select_runbook(hypotheses, situation, store, selector) → (Playbook, score, source) |
Rules-first, semantic-fallback: the keyword rules (surface_runbook) run first; when none fires, a RunbookSelector ranks the registered playbooks by embedding similarity of their symptoms field and picks the best above a threshold (source = rule / semantic / none). |
Adds some real intelligence to selection — retrieval among vetted playbooks, never an LLM choosing — catching semantically-obvious matches the keyword rules miss (ADR-026). Default off → rules-only. |
governance registry, RunbookSelector |
explain(hypothesis, context, situation) → str |
Produces human-readable advisory text for the top hypothesis via an ExplanationProvider (ADR-019) — set on RootCauseHypothesis.explanation, after ranking, so it can never affect confidence/order/runbook. |
Gives an on-call engineer plain-language context, on by default, with zero CI/test network dependency. | ExplanationProvider: TemplateExplanationProvider (deterministic, no network — used whenever llm_explanation_endpoint is unset) or OpenAICompatibleExplanationProvider (sync httpx POST to {endpoint}/chat/completions; any failure falls back to the template) |
| Function | What it does | Why | Depends on |
|---|---|---|---|
select_playbook(situation) → Playbook |
Matches a diagnosed situation to a remediation via match_rule. |
Connects "what's wrong" to "what to do." | governance registry |
_denylist_reason(playbook) → reason | None |
Gate (before the sandbox): refuses dangerous step shapes (unsafe scale-to-zero, implausible limits, defeated probe, indeterminate revision), fail-closed. | Defense-in-depth over the closed action Literal — guards allowed verbs' shapes, and AI-authored drafts (ADR-024). |
— |
sandbox.rehearse(situation, plan) → PreflightResult |
Pre-flight (before approval): clones the target into a throwaway namespace, applies the same plan, watches the clone recover (pod-readiness, plus the same per-metric predicate as verify_health() on the post-fix check), tears it down. Honest caveat: this repo's static-scrape Prometheus setup doesn't independently scrape the clone's namespace, so the per-metric query typically returns None and fails safe — the rehearsal effectively decides on pod-readiness today, wiring that activates fully once per-clone scraping exists (ADR-029). |
Try the fix safely first — a failed rehearsal blocks an auto playbook and advises a hitl human (ADR-023). Fail-safe; off by default. |
Sandbox |
request_approval(playbook, situation) → decision |
Synchronous call to governance for RBAC + HITL approval — the pre-flight verdict rides on the request. | The structural HITL gate — action can't proceed without a yes (ADR-003). | governance-service |
execute(playbook) |
Runs the playbook's typed steps through a Remediator — now 7 Deployment-scoped verbs (restart/scale/rollback_deploy/wait/patch_resource_limits/rollback_to_revision/patch_probe), the Literal still closed (ADR-024). |
Performs the actual fix. | Remediator |
verify_health() → bool |
Checks system health after acting: pod-readiness AND, in the health_check_mode=k8s path, per-metric recovery. The metric half no longer always queries cpu_usage — it re-applies the ADR-027 DetectionPolicy to the metric(s) that actually FIRED (detect-and-verify symmetry: a metric is recovered when it's no longer anomalous by the rule that detected it), using the baseline the Situation already carries. Every firing metric must recover — a partial fix isn't healthy. Any query failure, or a score-only metric with no usable baseline, fails safe to not-recovered. Dry-run (health_check_mode=always, AlwaysHealthyChecker) is unaffected. |
Confirms the right thing recovered before declaring success — a memory/error/latency fix is no longer "verified" against cpu (ADR-029). | telemetry / health checks, services/correlation/detection_policy.DetectionPolicy |
rollback(playbook) |
Runs rollback_steps if health verification fails — including a fail-safe not-recovered verdict from the per-metric check above. |
Enforces reversible-only automation (ADR-007). | Remediator |
emit_outcome(...) |
Publishes a RemediationOutcome to remediation.outcomes. |
Feeds the feedback loop. | BusClient |
| Function | What it does | Why | Depends on |
|---|---|---|---|
check_rbac(actor, action, resource) → allow | deny |
Decides whether an actor may perform an action. | Every automated action passes here — the enforceable RBAC guarantee. | RBAC policy |
create_approval_request() → ApprovalRequest |
Opens a pending approval for a hitl playbook. |
Materializes the human decision point. | contracts.ApprovalRequest |
decide(id, approve | reject) |
REST endpoint (POST /approvals/{id}/decide) a human/console calls to approve or reject. Only approved/rejected are accepted (422 otherwise), the decider needs the matching approve/reject permission, a request that is no longer pending is a 409, and the decision is written to the audit log under the human's name. |
The approval interface the React console's Approve/Reject buttons drive. | RBAC, AuditSink |
expire(id) |
POST /approvals/{id}/expire — the requester (action-service) marks a request it stopped waiting for as expired at its HITL timeout. |
Without it a timed-out request stayed pending forever and the console offered an Approve that could no longer do anything. |
AuditSink |
get_approval(id) / list_approvals() |
REST reads (GET /approvals/{id}, GET /approvals) of the pending queue. |
Lets the HTTP gate poll for a decision across containers, and the dashboard show what's pending. | — |
write_audit(record) |
Appends an immutable AuditRecord. |
The compliance backbone (NIST AI RMF / DORA / EU AI Act). | AuditSink |
register_playbook() / list_playbooks() |
Maintains the CoE playbook registry. POST /playbooks?registered_by= needs the register permission (coe-admin), refuses hitl_mode=auto unless the actor may also graduate, and is audited. |
Standardization — playbooks are shared, not reinvented per team. A direct write bypasses the AI-proposal review, so it is an admin action, not an open one. | playbook store, RBAC |
graduate(id) / demote(id) |
POST /playbooks/{id}/graduate (hitl → auto) and POST /playbooks/{id}/demote (auto → hitl), both behind the graduate permission and audited. feedback-service calls graduate on clean real evidence (dry-run successes don't count unless GRADUATION_COUNT_SIMULATED) and demote when an auto playbook fails or rolls back. |
Automation expands on evidence and contracts on failure. | RBAC, playbook store, AuditSink |
propose_playbook(situation) → ProposedPlaybook |
POST /playbooks/proposed: calls a RunbookAuthor (LLM) to draft a typed runbook for a gap, forces hitl_mode=HITL + a server-assigned id, stores it as a proposal (not the live registry); 422 if the author declines. |
The AI proposes a runbook for a gap; model_validate rejects any unsafe draft (ADR-025). |
RunbookAuthor, proposed store |
approve_proposed(id) / reject_proposed(id) |
POST /playbooks/proposed/{id}/approve|reject — RBAC-gated (reuses approve/reject), audited. Approve registers the inner playbook into the live registry; reject does not. |
The human disposes — the only path from an AI draft to the live registry (ADR-025). | RBAC, playbook store, AuditSink |
| Function | What it does | Why | Depends on |
|---|---|---|---|
label_outcome(outcome) → TrainingRecord |
Turns a RemediationOutcome into a labeled training example (worked / failed / rolled back). Escalations never get here — the consumer drops result=escalated before the store write, because nothing was attempted and it is therefore no evidence about any runbook. |
Converts operational results into learning signal — the innovation. | contracts.RemediationOutcome |
persist(record) |
Writes the labeled record to the training store correlation-service reads. |
The physical link that closes the loop. | training store |
compute_metrics() |
Tracks success/rollback/failure rates and per-signature reliability from outcomes, over attempted remediations only (escalations are excluded from the denominator). | Proves remediation quality with real numbers. (True MTTR/MTTD, which need detection→resolution timestamps, are computed by the read-service — see §5.7.) | outcome + situation history |
The one service that sees a situation's whole lifecycle (first_seen → outcome ts), so it is
where live KPIs are computed truthfully. Holds no source-of-truth state — a rebuildable
projection of the event streams (ADR-009).
| Function | What it does | Why | Depends on |
|---|---|---|---|
apply_detected/diagnosed/outcome/suppressed(...) |
Folds each event into an in-memory projection keyed by situation id, mapping backend contracts to the exact shapes the React types expect; each call ends by publish()-ing a {"type": "changed"} nudge to any subscribed SSE clients. |
One place owns the read shape, so the UI needs no translation layer. | the four situation/outcome topics |
situations(now_ms) / outcomes() |
Serve the live incident queue and outcome ticker (GET /situations, GET /outcomes). Ages out resolved/failed situations past a TTL and caps the total; active situations are never pruned. |
Keeps the queue legible over long runs without touching the durable event log. | the projection |
metrics() |
Computes the 8 dashboard KPIs — noise-reduction, real MTTR (mean of outcome.ts − first_seen over resolved situations), auto-remediated %, success rate, open/pending counts, suppressed count (GET /metrics). |
The Overview tiles read this — every number is derived, nothing fabricated. | the projection |
subscribe() / unsubscribe() / publish() |
Backs GET /stream (SSE): each connection gets its own bounded asyncio.Queue; consumer threads hand off delivery via loop.call_soon_threadsafe(...) (the thread-safe bridge onto the uvicorn event loop); a full queue drops the oldest event rather than blocking. |
Pushes changes to the console in near-real-time without the projection needing per-event serialization or a new dependency (ADR-018). | the projection, the uvicorn event loop |
reset() |
Clears the projection (POST /reset). Paired with correlation's POST /reset-baseline and demo-app's /fix (see scripts/reset.sh). |
A one-command clean slate for repeatable simulation runs — a simulation control, not a production endpoint. | the projection |
Scenario A — noise gets collapsed (Phase 1). A deploy triggers 200 correlated alerts.
ingestion normalizes and dedups them; correlation.detect_anomaly flags the abnormal ones;
correlation.correlate clusters them into one Situation on situations.detected. A
responder sees a single incident instead of 200 pages. No action is taken — Phase 1 stops at
noise reduction.
Scenario B — a full closed loop (Phase 3–4). The same Situation is diagnosed by rca
(top hypothesis: a crash-looping pod after the deploy; runbook: restart-pod). action.select_playbook
picks restart-pod (mode hitl, reversible). action.request_approval blocks on
governance; an on-call engineer POSTs approve to governance.decide. action.execute
restarts the pod, verify_health passes, emit_outcome reports success. feedback.label_outcome
records it; after enough successes, governance graduates restart-pod to auto
(ADR-008). On the next
retrain, correlation has learned this signature-plus-fix pair.
The functions above were delivered in slices mapped to the proposal's Phase 0–4. All four slices are complete — each is a working vertical.
| Slice | Proposal phase | What runs at the end of it | Status |
|---|---|---|---|
| 0 | — | Skeleton: common/ contracts + interfaces, Redis bus, docker-compose up, health endpoints on all six services. |
✅ |
| 1 | Phase 1 | ingestion → correlation: real telemetry in, Situation out. (Scenario A works.) |
✅ |
| 2 | Phase 2 | rca-service enrichment + hypotheses; governance audit + RBAC live. |
✅ |
| 3 | Phase 3 | action-service runs one hitl reversible playbook end to end with approval + rollback. (Scenario B works.) |
✅ |
| 4 | Phase 4 | feedback-service closes the loop; metrics; first playbook graduated to auto. |
✅ |
Since the original slices, three things were added that the first draft didn't design: the read-service (§5.7), the React operator console, and a live, breakable demo stack (a demo target app + real Prometheus + a scenario-reset script) so the whole loop can be driven and re-driven on docker-compose. See §8.
What's real, live, end-to-end today (verified on docker compose up):
- The full six-service closed loop, plus the read-service and the React console.
- Real Prometheus scraping a breakable demo app; anomalies detected by the online correlator; situations diagnosed; the HITL approval gate working across containers; a human approves in the console; the loop resolves; live KPIs (real MTTR, noise-reduction) populate.
- Repeatable simulations via
scripts/reset.sh(recover demo, forget detector baseline, empty the read model — no docker restart).
Persistence is real — behind a backend switch. The audit log, playbook registry, and
training store can be backed by Postgres (STORE_BACKEND=postgres), which is the compose
default: a postgres:16-alpine service, Alembic migrations applied by a one-shot migrate
step, and a hybrid schema (indexed columns + a JSONB payload that is the source of truth).
The same switch also makes two pieces of live runtime state durable — pending HITL
approvals and the correlator's z-score baseline — so a restart mid-incident resumes rather
than forgetting (approvals propagate errors like the audit log; the baseline snapshot/reload is
best-effort and fail-safe). file stays the default for tests and quick dev without Docker. See
docs/PERSISTENCE.md,
ADR-014, and
ADR-015.
Remediation is real — on an opt-in kind cluster. Behind REMEDIATOR_MODE=k8s /
HEALTH_CHECK_MODE=k8s, action-service drives a real Kubernetes API: KubernetesRemediator
restarts, scales, and rolls back an actual deployment via typed AppsV1Api calls (no shell, no
string parsing, never deletes), and KubernetesHealthChecker confirms recovery from pod
readiness plus a live Prometheus query, polled to a timeout. "Resolved" in this mode means the
pod was actually restarted. This is a documented runbook against a local kind cluster —
deploy/k8s/README.md — not part of CI; CI/pytest bind fake K8s
clients and never touch a real cluster.
Dry-run stays the default everywhere else (compose without the k8s overlay, tests, CI):
REMEDIATOR_MODE defaults to dry_run — DryRunRemediator logs the steps and
AlwaysHealthyChecker reports success, so "resolved" means the fix was logged and simulated,
no real infrastructure touched (ADR-007).
Observability & readiness are wired uniformly. Every service emits structured logs
through the shared app factory (INTELLIOPS_LOG_FORMAT=text|json, default text; the compose
stack sets json — one JSON object per line for aggregation) and exposes two probes: /health
(liveness — always 200 while the process can answer, checks nothing external) and /ready
(readiness — actively pings the bus, plus Postgres for the DB-backed services, returning
200 {ready:true} or 503 {ready:false, failed:[...]} naming the down dependency). Compose runs
the /ready probe as a per-service healthcheck; a real cluster should map livenessProbe: /health
readinessProbe: /ready(ADR-016, docs/OBSERVABILITY.md).
The console is real-time, with a live pipeline view. The read-service pushes over SSE
(GET /stream, a stdlib thread→async pub/sub inside ReadModel) so the console re-fetches within
about a second of a backend change, falling back to the previous 5-second poll if the stream can't
be used (ADR-018). The console gained
two tabs: Pipeline, which animates every open incident through five stage lanes as its status
changes, and Audit, a filterable explorer over the audit trail. Overview's fleet health and
sparklines are now derived from real live data rather than falling back to mock values, and the
whole console was repainted to Apple's light website palette. See docs/UI.md.
Detection and RCA got measurably smarter. A CORRELATOR_KIND=river|robust|trained
switch (ADR-019)
adds a seasonal robust-z-score detector (robust) and a persisted, re-fittable
scikit-learn IsolationForest blended on top of it (trained) behind the unchanged
river default; POST /retrain is the real fit trigger (not automatic — the
feature buffer is empty at boot). RCA's rank_hypotheses gained an optional
reliability-weighted boost from real remediation outcomes, and every diagnosed
hypothesis now carries an on-by-default advisory explanation (template, or a
configured OpenAI-compatible endpoint with a template fallback on any error) that
never affects ranking or the suggested runbook. A reproducible, seeded benchmark
(scripts/benchmark.py → docs/BENCHMARKS.md) measures the
actual gains and the actual trade-offs (higher recall but also higher
false-positive rate on robust/trained) against the river baseline, with one
comparison CI-enforced.
A real sample system now drives the pipeline. Meridian (services/meridian/) — four
backend services plus a client-portal/ops-panel UI, built on the same services.base.create_app
scaffold — runs alongside IntelliOps in docker compose up. Each service emits a USE+RED metric
set (11 gauges — CPU, memory, disk, saturation, queue depth, DB-pool utilization, request rate,
error rate, p50/p99 latency) and accepts 8 typed fault scenarios (saturation, latency,
error, memory_leak, traffic_surge, dependency_outage, db_exhaustion, crash), each
moving a realistic metric cluster rather than a single gauge — error/dependency_outage
deliberately hold cpu_usage at baseline so RCA doesn't misdiagnose an error incident as capacity.
It is wired in additively: a Prometheus scrape job per service, the ingestion query broadened to an
11-name regex selector in the compose environment only (the code default is unchanged), and a
shared volume that lets rca-service see Meridian's deploy markers for the first time. Three fault
scenarios were run sequentially against real Docker and each produced the expected, distinct
diagnosis — scale-service, restart-pod, rollback-deploy — through the unmodified
detection/RCA/action path; the newer scenarios' metric families don't yet have dedicated RCA rules
(that mapping is Phase 3 of the metrics arc), so they detect but may not richly diagnose today.
Faults must be injected one at a time: correlation-service groups anomalies by time window, not
by service, so concurrent faults on two Meridian services would merge into a single Situation — a
real constraint, confirmed live, that the Meridian UI enforces with a sequential-injection guard.
See docs/MERIDIAN.md and
ADR-020.
What is still deliberately simulated / deferred:
- Auth is a config-switched edge gate (ADR-017).
Under
AUTH_MODE=tokenthe console authenticates with a shared bearer token (VITE_AUTH_TOKEN) and the read/console endpoints are gated — there's no public read surface. The/reset,/reset-baseline,/reset-approvals,/break, and/fixendpoints are still simulation controls (gated under token mode, but must be removed or gated when this stack is pointed at a real system)./reset-baselineand/reset-approvalsnow clear the runtime-state DB tables (correlation_baseline,approvals) while preserving the audit and training records. - Automated model retraining.
POST /retrainfits and persists thetrainedcorrelator's model on demand; nothing schedules or triggers it automatically yet (see ADR-019).robust's per-hour seasonal windows are also in-process only — not yet a durable table — so a restart re-warms them from scratch. - Demo/report polish, per WORKPLAN.md.