Harden outbox, worker leases, and dashboard; publish canonical benchmark evidence

.github/workflows/ci.yml

@@ -12,7 +12,13 @@ jobs:
         with: {go-version: "1.25.x", cache: true}
       - run: test -z "$(gofmt -l cmd internal sdk/go gen tests/integration)"
       - run: go vet ./...
-      - run: go test -race -coverprofile=coverage.out ./cmd/... ./internal/... ./sdk/go/...
+      - run: go test -race -covermode=atomic -coverprofile=coverage.out ./cmd/... ./internal/... ./sdk/go/...
+      - run: scripts/check-go-coverage.sh coverage.out 10
+      - run: go test -coverprofile=workflow-coverage.out ./internal/workflow && scripts/check-go-coverage.sh workflow-coverage.out 70
+      - run: go test -coverprofile=auth-coverage.out ./internal/auth && scripts/check-go-coverage.sh auth-coverage.out 40
+      - run: go test -coverprofile=live-coverage.out ./internal/live && scripts/check-go-coverage.sh live-coverage.out 70
+      - uses: actions/upload-artifact@v4
+        with: {name: go-coverage, path: "*-coverage.out"}
   python:
     runs-on: ubuntu-latest
     steps:
@@ -22,6 +28,8 @@ jobs:
       - run: pip install -e 'sdk/python[dev]'
       - run: ruff check sdk/python examples
       - run: cd sdk/python && mypy runmesh && pytest --junitxml=../../pytest.xml
+      - uses: actions/upload-artifact@v4
+        with: {name: python-coverage, path: sdk/python/coverage.xml}
   web:
     runs-on: ubuntu-latest
     defaults: {run: {working-directory: web}}
@@ -32,8 +40,10 @@ jobs:
       - run: npm ci
       - run: npm run lint
       - run: npm run build
-      - run: npm test -- --run
+      - run: npm run test:coverage
       - run: npm audit --audit-level=high
+      - uses: actions/upload-artifact@v4
+        with: {name: web-coverage, path: web/coverage/coverage-summary.json}
   protobuf:
     runs-on: ubuntu-latest
     steps:
@@ -68,16 +78,29 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-go@v5
         with: {go-version: "1.25.x", cache: true}
-      - run: go test -tags=integration -count=1 -timeout=10m ./tests/integration
+      - run: go test -tags=integration -count=1 -timeout=15m ./tests/integration
   containers:
     runs-on: ubuntu-latest
     needs: [go, python, web, protobuf, integration]
-    strategy: {matrix: {service: [control-plane, scheduler, worker-go]}}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - {service: control-plane, dockerfile: Dockerfile, build-args: "SERVICE=control-plane"}
+          - {service: scheduler, dockerfile: Dockerfile, build-args: "SERVICE=scheduler"}
+          - {service: worker-go, dockerfile: Dockerfile, build-args: "SERVICE=worker-go"}
+          - {service: worker-python, dockerfile: sdk/python/Dockerfile, build-args: ""}
+          - {service: web, dockerfile: web/Dockerfile, build-args: "WEB_DEVELOPMENT_MODE=false"}
     steps:
       - uses: actions/checkout@v4
       - uses: docker/setup-buildx-action@v3
       - uses: docker/build-push-action@v6
-        with: {context: ., build-args: "SERVICE=${{ matrix.service }}", tags: "runmesh/${{ matrix.service }}:${{ github.sha }}", load: true}
+        with:
+          context: .
+          file: ${{ matrix.dockerfile }}
+          build-args: ${{ matrix.build-args }}
+          tags: runmesh/${{ matrix.service }}:${{ github.sha }}
+          load: true
       - uses: aquasecurity/trivy-action@v0.36.0
         with: {image-ref: "runmesh/${{ matrix.service }}:${{ github.sha }}", severity: CRITICAL, exit-code: "1", ignore-unfixed: true}
   kind-upgrade:

.github/workflows/release.yml

@@ -33,7 +33,7 @@ jobs:
             build-args: ""
           - component: web
             dockerfile: web/Dockerfile
-            build-args: VITE_DEV_AUTH=false
+            build-args: WEB_DEVELOPMENT_MODE=false
     steps:
       - uses: actions/checkout@v6
       - uses: docker/setup-qemu-action@v4

.gitignore

@@ -1,7 +1,17 @@
 .env
+.env.*
+!.env.example
+.secrets/
+*.pem
+*.key
+*.p12
+credentials.json
 .DS_Store
 bin/
 coverage/
+.coverage
+coverage.xml
+htmlcov/
 dist/
 node_modules/
 __pycache__/

CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+All notable changes are documented here. This project follows Semantic Versioning.
+
+## [0.1.0] - Unreleased
+
+### Added
+
+- Multi-tenant workflow and DAG lifecycle APIs with OIDC and scoped API-key authentication.
+- Go and Python workers using Kafka delivery, expiring leases, fenced mutations, retries, cancellation, and dead-letter replay.
+- S3-compatible input, output, and log artifacts with presigned transfer URLs.
+- Per-tenant Redis rate limiting and live WebSocket notifications with a polling fallback.
+- Prometheus metrics, OpenTelemetry traces, structured logs, and six Grafana dashboards.
+- Docker Compose, Helm, validated AWS Terraform, browser tests, chaos tests, and reproducible local benchmark tooling.
+
+### Reliability
+
+- Transactional outbox publication now claims work in short PostgreSQL transactions and publishes outside row locks.
+- Workflow-run Kafka keys preserve ordering across multiple partitions and concurrent publishers.
+- Worker operations verify tenant ownership, worker identity, active state, and unexpired leases.
+- Fixed Kafka/Redpanda topic initialization on a completely fresh stack, where `rpk topic describe` could report success before the topic actually existed.
+- Fixed heterogeneous Go/Python worker capability routing on a shared consumer group.
+
+### Known limitations
+
+- Benchmark evidence is local and does not establish production capacity.
+- The Terraform configuration is validated with mock providers but has not been applied to AWS.
+- Kafka is externally supplied; the Terraform configuration does not create MSK.
+- At-least-once execution cannot make arbitrary handler side effects exactly-once.
+- `v0.1.0` images and release artifacts have not been published.

CONTRIBUTING.md

@@ -0,0 +1,120 @@
+# Contributing to RunMesh
+
+Thank you for helping improve RunMesh. Changes should preserve tenant isolation,
+at-least-once delivery, per-workflow ordering, and backward-compatible protobuf
+evolution.
+
+## Local setup
+
+The shortest path to a complete development environment is Docker Engine or
+Docker Desktop with Docker Compose v2:
+
+```bash
+git clone https://github.com/samarth1412/RunMesh.git
+cd RunMesh
+docker compose up --build -d
+docker compose ps
+bash tests/e2e/smoke.sh
+```
+
+For development outside containers, install Go 1.25, Python 3.11 or newer, and
+Node.js 22. Then install the Python and web dependencies:
+
+```bash
+python3 -m pip install -e 'sdk/python[dev]'
+npm --prefix web ci
+```
+
+## Testing
+
+Run focused tests while developing and the relevant full suite before opening a
+pull request:
+
+```bash
+gofmt -w cmd internal sdk/go gen tests/integration
+go vet ./...
+go test -race ./cmd/... ./internal/... ./sdk/go/...
+go test -coverprofile=workflow-coverage.out ./internal/workflow
+scripts/check-go-coverage.sh workflow-coverage.out 70
+
+ruff check sdk/python examples
+(cd sdk/python && mypy runmesh && pytest) # includes a 40% worker-SDK floor
+
+npm --prefix web run lint
+npm --prefix web run build
+npm --prefix web run test:coverage
+
+buf lint
+go test -tags=integration -count=1 -timeout=20m ./tests/integration
+```
+
+Integration tests use Docker through Testcontainers. The browser suite expects
+the Compose stack:
+
+```bash
+docker compose up -d --build
+npm --prefix web run test:e2e
+```
+
+CI additionally checks protobuf compatibility, container vulnerabilities,
+Terraform, Helm, rendered Kubernetes resources, the Compose smoke path, and a
+kind rolling upgrade.
+
+## Protobuf changes
+
+Only make additive, backward-compatible changes to `proto/`. Regenerate both Go
+and Python clients and commit them in the same pull request:
+
+```bash
+buf generate
+buf lint
+buf breaking --against '.git#branch=origin/main'
+```
+
+If `buf` is not installed locally, use the pinned container:
+
+```bash
+docker run --rm -v "$PWD:/workspace" -w /workspace \
+  bufbuild/buf:1.47.2 generate
+```
+
+## Database migrations
+
+Schema changes require paired, reversible migrations. Create the next numbered
+`.up.sql` and `.down.sql` files under `migrations/`, use idempotent DDL where it
+is safe, and test both directions against disposable data. Apply pending local
+migrations with:
+
+```bash
+make migrate
+```
+
+Never edit a migration that may already have been applied; add a new migration.
+
+## Benchmarks and failure tests
+
+Performance claims require raw, machine-readable evidence tied to a source
+commit. Start from a disposable Compose environment and run:
+
+```bash
+tests/benchmarks/run-local.sh tests/benchmarks/results/YYYY-MM-DDTHHMMSSZ
+```
+
+The benchmark driver starts the local stack, stops schedulers, scales and terminates workers, and writes
+into a new result directory. It refuses to overwrite existing evidence. Do not point it at shared or production
+infrastructure. Do not hand-edit raw output. Record failed runs honestly, and
+distinguish API throughput, scheduling throughput, and end-to-end completion
+throughput. See [`docs/benchmarks.md`](docs/benchmarks.md).
+
+## Pull requests
+
+- Keep changes focused and explain the behavior or failure mode being changed.
+- Add tests for new state transitions, authorization boundaries, and recovery
+  behavior.
+- Update OpenAPI, protobuf clients, deployment configuration, and documentation
+  with the behavior they describe.
+- Include exact validation commands and results. Do not describe a partial test
+  run as a full validation.
+- Do not include secrets, local environment files, generated credentials, or
+  unreviewed benchmark output.
+- Preserve existing commit attribution and do not rewrite shared history.

Dockerfile

@@ -1,7 +1,7 @@
 FROM golang:1.25-alpine AS build
 WORKDIR /src
 RUN apk add --no-cache ca-certificates git
-COPY go.mod ./
+COPY go.mod go.sum ./
 RUN go mod download
 COPY . .
 ARG SERVICE=control-plane

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Samarth Vinayaka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -10,8 +10,18 @@
 
 RunMesh stores versioned DAGs, schedules dependency-aware tasks, and delivers work with at-least-once semantics. PostgreSQL is the source of truth, a transactional outbox bridges state to Kafka/Redpanda, and expiring leases let work recover safely after worker or infrastructure failures.
 
+| See it | Run it | Verify it |
+| --- | --- | --- |
+| [Dashboard demo](#demo) | `docker compose up --build -d` | [Raw benchmark evidence](tests/benchmarks/results/) |
+
 It ships as a complete local platform: Go control plane and scheduler, Go and Python workers, a React operations dashboard, OIDC authentication, tenant-scoped API keys, S3-compatible artifacts, live updates, metrics, logs, traces, Helm charts, and validated AWS Terraform.
 
+Latest local evidence on an Apple M4 / 16 GiB machine: **407.403 scheduler
+dispatches/second across 10,000 tasks**, **100.0 authenticated reads/second
+at 3.307 ms p95**, and **10,000/10,000 tasks completed with zero permanent
+loss after all 20 workers were terminated**. These are reproducible local
+Docker results, not production-capacity claims.
+
 ## Highlights
 
 - **Durable execution** — transactional run creation, dependency-aware scheduling, fenced state transitions, retries, cancellation, and dead-letter replay.
@@ -49,6 +59,18 @@ PostgreSQL remains authoritative throughout the lifecycle. Kafka delivery can be
 
 Read the deeper [architecture](docs/architecture.md), [delivery semantics](docs/delivery-semantics.md), and [failure-mode analysis](docs/failure-modes.md).
 
+## Demo
+
+The dashboard is exercised against the real Compose stack in Playwright; it does not use a mock API. The [one-minute recovery video](docs/demo/runmesh-recovery.webm) creates and runs a DAG, terminates both bundled workers during an active lease, and shows the successful recovered attempt.
+
+| Operations overview | Completed dependency graph |
+| --- | --- |
+| ![RunMesh operations overview](docs/demo/dashboard-overview.png) | ![RunMesh completed run detail](docs/demo/run-detail.png) |
+
+![RunMesh lease recovery with immutable attempt history](docs/demo/lease-recovery.png)
+
+The exact capture procedure is documented with the media in [`docs/demo`](docs/demo/README.md); no mock API or generated screenshot is used.
+
 ## Quick start
 
 You need Docker Engine/Desktop with Docker Compose v2, `curl`, and Python 3.
@@ -137,6 +159,15 @@ worker.run()
 
 See the complete [Python worker example](examples/python_worker.py) and the [artifact guide](docs/artifacts.md).
 
+The canonical Go SDK import is:
+
+```go
+import runmesh "github.com/samarth1412/RunMesh/sdk/go"
+
+client := runmesh.NewClient("localhost:7001", "rm_<id>_<secret>", "worker-1")
+defer client.Close()
+```
+
 ## Reliability model
 
 RunMesh deliberately promises **at-least-once**, not exactly-once, execution. Task handlers that perform side effects should deduplicate using the stable `context.idempotency_key`.
@@ -169,13 +200,14 @@ The repository includes unit, race, integration, browser, security, chaos, deplo
 
 | Scenario | Recorded result |
 | --- | ---: |
-| Authenticated API load | 100.017 req/s, 4.316 ms p95, 0 failures |
-| Simultaneously active workflows | 1,000 |
-| Scheduler dispatch | 196.078 tasks/s across 10,000 tasks |
-| Worker termination and recovery | 10,000/10,000 succeeded, 0 lost |
+| Authenticated API reads | 100.0 req/s; 1.701/3.307/4.657 ms p50/p95/p99; 0 failures |
+| Workflow submissions | 50.028 req/s; 1.587/7.012/220.024 ms p50/p95/p99; 0 failures |
+| Simultaneously active workflows | 1,000 created; 0 failures; 57.608 tasks/s drain |
+| Scheduler dispatch | 407.403 tasks/s across 10,000 tasks |
+| Worker termination and recovery | 20 workers terminated; 10,000/10,000 succeeded; 7 recovered attempts; 0 lost |
 | Duplicate submission and delivery | Passed |
 
-These numbers are transparent local evidence, not a universal capacity claim. Hardware details, commands, image digests, source SHA, and raw machine-readable output are committed in the [2026-07-20 benchmark results](tests/benchmarks/results/2026-07-20/README.md).
+These numbers are transparent local evidence, not a universal capacity claim. Hardware details, commands, image digests, source SHA, and raw machine-readable output are committed in the [2026-07-21T003600Z benchmark results](tests/benchmarks/results/2026-07-21T003600Z/README.md), the canonical run against the current `HEAD`. The complete [local validation ledger](docs/validation.md) separates executed checks from remote or cloud work that remains unverified.
 
 ## Development
 
@@ -205,6 +237,22 @@ make lint             # format/check Go, Python, and TypeScript
 
 CI also validates protobuf compatibility, container vulnerability scans, Terraform, Helm, rendered Kubernetes manifests, Playwright flows, and a live kind rolling upgrade.
 
+Coverage gates intentionally target risk rather than 100%: the workflow state machine, authentication, live event broker, Python worker, and dashboard each have documented minimums in CI. Docker-backed integration tests cover PostgreSQL scheduler and storage behavior.
+
+## Limitations and future work
+
+- Published performance evidence is from one local Docker Desktop machine, not a production capacity guarantee.
+- The AWS Terraform is formatted, validated, and tested with mock providers; it has not been applied and no AWS deployment is claimed.
+- Useful worker parallelism is bounded by Kafka partition count, and a single workflow run remains intentionally confined to one partition.
+- Workers in the same Kafka consumer group must expose the same handler set; heterogeneous capability pools require separate group IDs.
+- The outbox removes broker I/O from database transactions, but PostgreSQL claim and acknowledgement writes still bound dispatch throughput.
+- The recorded 10,000-task crash run recovered all interrupted leases, but the
+  recovered-attempt delay was 132.797 seconds p50 under backlog; prioritizing
+  expired leases is future work.
+- At-least-once execution cannot make arbitrary handler side effects exactly-once. Handlers must use the stable task idempotency key.
+- Redis Pub/Sub notifications are best-effort; the dashboard re-reads durable state and falls back to polling.
+- The repository prepares `v0.1.0`, but no release or versioned image is published until a maintainer explicitly creates the tag.
+
 ## Deployment
 
 The Helm chart deploys independently scalable control-plane, scheduler, dashboard, and worker pools with probes, resource limits, autoscaling, disruption budgets, restricted security contexts, and default-deny NetworkPolicies.
@@ -236,4 +284,7 @@ Pushing a `v*` tag builds five multi-architecture GHCR images with semantic-vers
 - [Security model](docs/security.md)
 - [Artifact handling](docs/artifacts.md)
 - [Benchmarks](docs/benchmarks.md)
+- [Local validation report](docs/validation.md)
 - [Deployment and releases](docs/deployment.md)
+- [Architecture decisions](docs/adr/)
+- [Draft v0.1.0 release notes](docs/release-v0.1.0.md)