[P1] Establish production telemetry, SLOs, incident response, and disaster-recovery evidence

[seonghobae]

Production gap

Wardnet exposes health/KPI/support-bundle surfaces and has a local operations runbook, but production operations still lack a declared service-level model, correlated traces/logs/metrics, actionable alerting, tamper-evident audit retention, on-call and incident procedures, tested backup restoration, and measured recovery objectives. The existing runbook explicitly limits the baseline to local and controlled lab deployments.

Audited protected base: b53dc7a1b8904a16752abbdc04429df893a4e32e.

Required observability contract

1. Adopt vendor-neutral OpenTelemetry context propagation for ingress, gateway/upstream calls, database transactions, outbox workers, identity checks, destination-policy decisions, and integration calls.
2. Emit structured logs with timestamp, severity, service/version, tenant-safe correlation fields, trace/span IDs, route/policy IDs, decision/reason codes, and error class.
3. Redact credentials, authorization headers, cookies, sensitive query/body values, personal data, and threat-feed payloads by default; add tests for every logging surface.
4. Export low-cardinality metrics for request rate/errors/duration, block/monitor decisions, upstream saturation/failures, auth denials, destination-policy denials, database health/latency/pool, outbox backlog/age/retries/dead letters, limiter saturation, feed freshness, and deployment version.
5. Split liveness, readiness, and startup semantics. Readiness must depend on required authorities and schema compatibility but not on optional analytics backends.
6. Define trace sampling and security-event retention so incident evidence is sufficient without uncontrolled cost or sensitive-data accumulation.
7. Make audit records append-only/tamper-evident with retention, legal/privacy review, export, and access-control policies.

SLO and alert requirements

Define initial objectives and revise from measured capacity:

• data-plane availability and latency by route/traffic class;
• management-plane availability and policy propagation latency;
• enforcement-decision correctness/false-positive guardrails;
• threat-feed freshness;
• identity revocation propagation;
• outbox delivery age and dead-letter rate;
• backup RPO and restore RTO.

Alerts must use burn-rate/sustained-impact logic where appropriate, have an owner, severity, runbook link, deduplication, and testable notification path. Avoid alerts on raw high-cardinality events.

Incident response and recovery

1. Add severity classification, on-call ownership, triage, containment, evidence preservation, communications, regulatory/customer escalation inputs, recovery, and post-incident review.
2. Add focused runbooks for credential compromise, unauthorized policy change, false-positive mass block, SSRF/egress abuse, feed poisoning, database corruption, outbox backlog, dependency outage, signing/provenance failure, and rollback failure.
3. Declare backup cadence, retention, encryption, restore authority, RPO/RTO, and dependency-recovery order.
4. Automate isolated restore drills and production-shaped game days; retain evidence and corrective actions.
5. Link incident IDs to relevant traces, audit events, deployment digest, policy version, and external-effect receipts.

RED → GREEN verification

• End-to-end trace continuity across HTTP → policy → PostgreSQL → outbox → integration.
• Secret/PII canary values never appear in logs, traces, metrics, support bundles, or errors.
• Readiness fails for missing mandatory auth/database/schema dependencies and remains healthy for optional analytics outage as designed.
• Alert tests inject known SLO breaches and verify deduplicated notification plus runbook linkage.
• Restore drill reconstructs a production-shaped state and proves semantic invariants within RPO/RTO.
• Game-day scenarios exercise detection, containment, rollback, and evidence collection.

Acceptance criteria

• Operators can identify the affected tenant/route/policy/deployment and first causal boundary from one incident signal without secrets leakage.
• SLOs, error budgets, alert owners, and runbooks are version-controlled and tested.
• Liveness/readiness/startup semantics are production-correct.
• Backup restore and at least one full incident game day meet measured objectives.
• Post-incident corrective actions feed back into issues/tests and are traceable.

Dependencies

• [P0] Add an authoritative PostgreSQL control plane with tenant isolation and recoverable migrations #80 supplies the durable authority and recovery substrate.
• [P0] Add a transactional outbox and idempotent leased workers for external effects #81 supplies durable worker telemetry.
• [P1] Integrate Keyverse identity, tenant authorization, consent, and human approval evidence #82 supplies identity/audit context.
• [P1] Build an immutable signed release, promotion, and rollback pipeline #84 supplies immutable deployment identity and rollback evidence.

References

• Nelson, A., Rekhi, S., Souppaya, M., & Scarfone, K. (2025). Incident response recommendations and considerations for cybersecurity risk management: A CSF 2.0 community profile (NIST SP 800-61 Rev. 3). https://doi.org/10.6028/NIST.SP.800-61r3
• OpenTelemetry Authors. (2026). OpenTelemetry documentation. https://opentelemetry.io/docs/
• National Institute of Standards and Technology. (2022). Secure Software Development Framework (SSDF) version 1.1 (NIST SP 800-218). https://doi.org/10.6028/NIST.SP.800-218