KubePulse

Detects dangerous releases that pass readiness probes but are unsafe for users.

5 scenarios · 0 false-safe decisions.

Python · Kubernetes · FastAPI · Prometheus · Docker Compose

---

Why This Exists

Problem: Kubernetes readiness probes check whether a container is alive. They do not check whether the deployment is safe for users. A service can pass every probe while DNS is failing, p95 latency has tripled, and the error budget is at zero.

Impact: False-green deployments roll out. The problem compounds across deployment waves. Customers are impacted before anyone notices.

Proof: KubePulse compares probe state against user-visible health across four validation layers. Blocked 5 dangerous deployments in controlled testing — including a cascade where probes stayed green through +333% p95 latency and 8% error rate. 0 false-safe decisions.

---

Quick Links

• Latest report: reports/latest/quality_soak_report.json
• Tests: make test
• Demo: make demo

---

Latest Project Output

{
  "release_decision": "BLOCK",
  "probes_say_healthy": true,
  "safe_to_operate": false,
  "p95_latency_impact": "+333%",
  "error_rate_increase": "+8%",
  "error_budget_remaining": "0.0%",
  "rollback_recommended": true
}

Generated by CI. Source: reports/latest/quality_soak_report.json

---

Live Metrics Snapshot

5 scenarios · 0 false-safe decisions.

Signal	Value
Release Decision	BLOCK
Recovery Time	12s
P95 Latency Impact	+333%
Error Rate Increase	+8%

Generated from the latest report artifact in reports/latest/.

---

What This Proves

✓ SRE
✓ Production Engineering
✓ Kubernetes
✓ Reliability

---

Latest Release Safety Report

Scenario	Probe State	Safe to Operate	Decision
Readiness false positive	PASS	false	BLOCK
DNS failure	PASS	false	BLOCK
API latency injection (+22,831% p95)	PASS	false	BLOCK
Multi-service cascade (+333% p95)	PASS	false	BLOCK
AMD MI300X burst (+608% p95)	PASS	false	BLOCK

Report regenerated by CI. Source: reports/latest/safety_report.md

---

Why KubePulse

Kubernetes readiness probes check whether a container is alive. They do not check whether the deployment is safe for users.

A service passes every probe while downstream DNS is failing, p95 latency has tripled, and the error budget is at zero. The next deployment wave rolls out. The problem compounds.

KubePulse measures user-visible health across four validation layers, compares against probe state, and issues an explicit release decision:

Deployment rolls out. All readiness probes: PASS.

Behind the scenes:
  edge-service → api-service → auth-service → postgres
  auth-service connection pool exhausted under load
  Cascade: api-service timeout → edge-service retry storm

KubePulse Layer 1:  p95 latency +7,623% · error rate 8%
Layer 3:            error_budget_remaining: 0.0%
Layer 4:            probes_say_healthy: true · safe_to_operate: false

Release decision: BLOCK · rollback_recommended: true

---

Architecture

Deployment Trigger
      │
      ▼
Layer 1: Health Signals
  p50 · p95 · p99 · error rate
      │
      ▼
Layer 2: Network Validation
  DNS · TCP · TLS · auth
      │
      ▼
Layer 3: SLO Evaluation
  error budget · resilience score
      │
      ▼
Layer 4: Probe Integrity
  probes_say_healthy  vs  safe_to_operate
      │
      ├── divergence detected → release_decision: BLOCK + rollback recommendation
      └── all pass           → release_decision: continue

---

Repository Structure

KubePulse/
├── lab/network-lab/  Docker Compose scenarios + scripts
├── scenarios/        YAML scenario definitions
├── gate/             4-layer release-quality gate
├── diagnostics/      DNS · TCP · TLS · auth validation
├── docs/             architecture · case study
└── reports/          soak-test + quality gate outputs (CI-generated)

---

Run Locally

git clone https://github.com/kritibehl/KubePulse && cd KubePulse
docker compose -f lab/network-lab/docker-compose.yml up -d --build
make demo    # all 5 scenarios
make test    # 0 false-safe decisions
make report  # → reports/latest/quality_soak_report.json

---

Tests

make test
# ✓ readiness false positive     — probes=true · safe=false → BLOCK
# ✓ DNS failure                  — 0/25 requests succeed → BLOCK
# ✓ API latency injection        — p95 +22,831% → BLOCK
# ✓ multi-service cascade        — resilience score 100→46 → BLOCK
# ✓ AMD MI300X burst             — p95 +608% → BLOCK
# 5 scenarios · 0 false-safe decisions

---

Selected Outputs

Multi-service cascade scenario:

{
  "probes_say_healthy": true,
  "safe_to_operate": false,
  "release_decision": "block",
  "error_budget_remaining": "0.0%",
  "what_probes_missed": "333% p95 drift · 8% error rate · 9% availability gap",
  "rollback_recommended": true
}

Operations Proof: Release Gate, Alerts, and Incident Output

KubePulse includes operational artifacts for deployment and reliability workflows:

• CI/CD release gate demo
• monitoring and alerting summary
• service health dashboard summary
• incident report output
• incident response workflow

Artifacts:

• ci_release_gate/release_gate_result.json
• ci_release_gate/release_gate_report.md
• monitoring_alerting/alert_summary.json
• monitoring_alerting/alert_policy.md
• monitoring_alerting/service_health_dashboard.md
• incident_reports/incident_001.json
• incident_reports/incident_001.md
• docs/incident_response_workflow.md

These artifacts show how KubePulse converts validation results into rollout gates, alerts, and incident reports.

SLO Dashboard Artifacts

KubePulse includes service-level objective artifacts for SRE and platform workflows:

• service_level_objectives/slo_report.md
• service_level_objectives/error_budget_policy.md
• service_level_objectives/burn_rate_alerts.md

Example SLO output:

{
  "availability": 99.92,
  "error_budget_remaining": 63,
  "release_decision": "block"
}

These artifacts map rollout decisions to availability, error-budget, burn-rate, latency and recovery-window signals.

Dataset SLA and Product Dashboard

KubePulse includes product-style reliability views for validation data and rollout operations.

Artifacts:

• dataset_sla/dataset_sla_report.json
• dataset_sla/dataset_sla_report.md
• product_dashboard/rollout_dashboard.json
• product_dashboard/rollout_dashboard.md
• product_dashboard/product_summary.md
• pipelines/export_product_dashboard.py

Dataset SLA tracks:

• freshness
• completeness
• pipeline latency
• failed jobs

Product dashboard tracks:

• rollout count
• approved releases
• blocked releases
• rollback-required releases
• incident categories
• time to detect

Example output:

{
  "availability": 99.92,
  "error_budget_remaining": 63,
  "rollout_count": 12,
  "blocked_releases": 4,
  "mean_time_to_detect_seconds": 18
}