OpsRehearse

Practice failures. Verify recovery.

OpsRehearse is a self-hosted platform for rehearsing IT failures, validating recovery procedures, and improving operational readiness. It is intended for IT support, operations, DevOps, and security practitioners who want to test a Runbook against a controlled failure instead of assuming that recovery will work.

The product follows one loop:

Prepare -> Rehearse -> Observe -> Diagnose -> Recover -> Verify -> Improve

The current milestone is the deterministic Help Desk target: a real support-request API, dedicated PostgreSQL database, PostgreSQL-backed notification queue, worker, correlated JSON logs, health checks, migrations, and a repeatable baseline reset contract. This target is the system that later OpsRehearse milestones will safely degrade, observe, recover, and verify.

What OpsRehearse Is

OpsRehearse is more than a sample web page. The planned system has three clear parts:

1. Control plane - guides an operator through an exercise, stores the timeline and evidence, versions Runbooks, verifies recovery, and produces an after-action report.
2. Isolated Runner - performs only registered, bounded actions against an approved exercise target. It will not provide arbitrary shell execution.
3. Exercise targets - realistic applications with observable failure and recovery behavior. The built-in Help Desk service is the first target.

The Help Desk target is deliberately small enough to understand but complete enough to exercise practical operations skills: API troubleshooting, PostgreSQL behavior, queue backlogs, worker health, structured logs, migrations, container operation, reset procedures, and recovery checks.

Current System

Employee
   |
   v
Help Desk API -----> PostgreSQL
                         |
                         v
                 notification_jobs
                         |
                         v
                  Help Desk Worker

Future exercise path:

OpsRehearse Control Plane -> Isolated Runner -> Help Desk target
             |                                      |
             +------ evidence and verification <----+

The repository currently supports the first diagram. The second diagram describes the approved product direction and is not implemented yet.

Current Boundary

This is an early local technical preview, not a production or enterprise release. It does not yet include the OpsRehearse control plane, fault Runner, authentication, external environments, or cloud adapters. The current UI belongs to the built-in Help Desk target; it is not the future exercise-operator console.

The first vertical slice is intentionally limited to one managed target and one bounded database connection-pool exhaustion scenario. Kubernetes, Kafka, microservices, cloud adapters, arbitrary shell execution, production write operations, SSO, multi-tenancy, and a plugin SDK are outside the v0.1 boundary.

Requirements

• Python 3.13
• uv
• Docker Desktop with Docker Compose

No API key or external account is required.

Verify the Complete Day 1 Path

make install
make verify

make verify removes the local target volume, builds the containers, starts PostgreSQL, applies migrations, starts the API and worker, waits for readiness, submits a support request, confirms notification completion, runs all tests, and checks migration drift.

Run Locally

make up

Open http://127.0.0.1:8080 to submit an employee support request.

Useful endpoints:

• GET /health/live
• GET /health/ready
• POST /api/requests
• GET /api/requests/{request_id}

Stop the services without deleting the database volume:

make down

Delete the local target database and return to a clean installation:

make down-clean

Development Checks

make test
HELPDESK_DATABASE_URL='postgresql+psycopg://helpdesk:helpdesk_local@127.0.0.1:55432/helpdesk' uv run alembic check

The default credential is a local Compose development value. Override POSTGRES_PASSWORD when needed and keep real credentials out of Git.

Documentation

• Product specification
• Architecture and trust boundaries
• v0.1 vertical slice
• Roadmap and release gates
• Threat model
• Risk register
• Architecture decisions

Long-Term Direction

Development is organized as verified vertical slices rather than a fixed-date feature list. The immediate objective is one complete exercise: establish a healthy Help Desk baseline, safely exhaust database connections, observe API degradation and worker backlog, follow a versioned Runbook, recover the target, automatically verify the result, reset the environment, and generate an evidence-backed after-action report.

After that loop is reliable, later milestones can deepen the platform through additional IT support and security scenarios, stronger local authorization, external observability integrations, team workflows, and carefully reviewed environment adapters. Bring-your-own-environment support, cloud integrations, Kubernetes support, an SDK, and AI-assisted analysis remain candidates rather than promises; each requires a separate product and security decision.

The durable goal is a practical operational-readiness lab where failure injection is bounded, recovery is measurable, and every conclusion can be traced to evidence.

Project Status

The deterministic Help Desk baseline is implemented. The control plane, isolated Runner, complete exercise workflow, and v0.1 release gate remain future work. See the roadmap for exact deliverables and acceptance criteria.