OHMF is a distributed real-time messaging backend written in Go. It accepts messages over an HTTP/WebSocket gateway, moves them through a Kafka event pipeline to background processors, and persists conversation state durably in PostgreSQL (with Cassandra wired in as a secondary message store). Redis carries presence and low-latency realtime fan-out. The whole stack runs locally under Docker Compose.
The project exists to be a concrete, inspectable example of the correctness problems that show up in a multi-service messaging system - idempotency under concurrent sends, per-conversation ordering across multiple workers, at-least-once delivery with duplicate suppression, and cross-pod presence - and to show those problems being solved with tests that pin the behavior down. It is a working development environment and a correctness case study, not a finished, production-operated system.
Historical local experiments at a configured 120 msg/sec reported matching accepted/Postgres/Cassandra counts and drained Kafka lag — all on a single-node local cluster, not production. Those artifacts predate current identity/content reconciliation and crash-window fixes. Core reliability hardening is implemented and regression-tested; a fresh benchmark and live fault-injection campaign remain pending. See Limitations and Benchmarks.
- End-to-end backend ownership — gateway, Kafka pipeline, message processors, persistence, and delivery in a single coherent system
- Distributed messaging pipeline design — at-least-once delivery with duplicate suppression, per-conversation ordering, and idempotency under concurrent retries
- Real-time WebSocket / API behavior — presence tracking, typing indicators, fan-out across pods via Redis pub/sub
- Historical load/failure evidence — named local artifacts cover scaled load, processor pod deletion/rebalance, and backlog recovery, but are count-based and do not validate the remediated crash windows
- Operational docs and reproducible workflows — local Docker Compose and Kubernetes (k3s/k3d) bring-up, CI pipeline, and documented limitations
Full evidence chain for the M4 validation milestone:
| Document | Contents |
|---|---|
| docs/claims-and-evidence.md | Per-claim status, artifact links, and supported claim boundary |
| docs/validation-matrix.md | Test-by-test pass/fail table across all stages |
| benchmarks/results/2026-06-09-processor-scaling-matrix/summary.json | Historical scaled load: matching counts and drained lag |
| benchmarks/results/2026-06-10-processor-pod-deletion-120msgsec/summary.json | Historical pod deletion/rebalance: count-based settlement; current fixes not rerun |
| benchmarks/results/2026-06-10-processor-backlog-recovery-120msgsec/summary.md | Historical backlog recovery: count-based settlement after lag drain |
All results are local single-node cluster only. See Limitations for the full boundary.
Most "chat app" projects stop at create-read-update-delete over a database. The interesting and hard part of messaging is everything around that: what happens when the same send is retried, when two workers fan out the same conversation concurrently, when a delivery event is redelivered, or when a user is connected to two pods at once. OHMF is built to make those failure paths explicit and to demonstrate the engineering - not just the happy path. The reliability work is documented with the specific test that proves each behavior in docs/reliability-hardening.md.
Recent hardening also covers privacy-sensitive paths that are easy to miss in ordinary feature
delivery: reaction history/export filtering, typing.stop privacy handling, websocket heartbeat
cleanup, cross-pod presence teardown, retryable fanout on Redis publish failure, persistent worker
failure surfacing, and per-conversation fanout ordering.
For a compact evidence trail, see docs/claims-and-evidence.md and the committed benchmark artifacts under benchmarks/results/. Those records show reliability and privacy hardening as first-class system behavior, not invisible maintenance.
What is actually present and used in the code (verified against go.mod files and the compose stack):
| Layer | Technology | Where |
|---|---|---|
| Language | Go 1.25 (multi-module) | ohmf/services/* |
| API / realtime | HTTP + WebSocket gateway (go-chi/chi, gorilla/websocket) |
ohmf/services/gateway |
| Event pipeline | Apache Kafka (segmentio/kafka-go) |
ohmf/services/gateway/internal/bus, processors |
| Authoritative store | PostgreSQL | gateway + processors |
| Secondary message store | Cassandra (gocql) - shadow-write enabled, reads off by default |
ohmf/services/gateway/internal/messages/cassandra_store.go |
| Presence / realtime fan-out | Redis | gateway, processors |
| Observability | Prometheus + Grafana dashboards | ohmf/infra/observability |
| Local orchestration | Docker Compose | docker-compose.yml, ohmf/infra/docker |
| Web client | Static HTML/CSS/vanilla JS | ohmf/apps/web |
| CI | GitHub Actions (OpenAPI validation, Go tests, docker build, containerized integration test) | .github/workflows |
Kubernetes: plain-Kubernetes (Kustomize, no Helm) manifests for local
single-node k3s/k3d validation live under deploy/k8s/. The repo now
contains a lighter smoke profile (overlays/local-k3s), a fuller local
pipeline profile (overlays/local-k3s-full), and an optional local gateway HPA
layer (overlays/local-k3s-full-hpa). These support local single-node evidence
only: they are not a production deployment package, not multi-node resilience
evidence, not Helm, and not benchmark-validated. See deploy/k8s/README.md.
Not currently in the repo: Helm charts and standalone WebSocket load-test client scripts (the benchmark load-generators run as Kubernetes Jobs, not standalone scripts). These are referenced as design intent only and are called out below.
WebSocket / HTTP clients
|
v
+----------------------+
| Gateway (API) | auth, validation, idempotency,
| chi + gorilla/ws | realtime fan-out
+----------------------+
| produce: msg.ingress.v1
v
+-------------+
| Kafka | partitioned topics + DLQs
+-------------+
| consume
v
+-----------------------------------+
| messages-processor | persist + retry semantics
+-----------------------------------+
| |
v v
+------------+ +------------+
| PostgreSQL | | Cassandra | (authoritative) (shadow write)
+------------+ +------------+
| produce: msg.persisted.v1
v
+-----------------------------------+
| delivery-processor / sms-processor| delivery receipts, SMS dispatch
+-----------------------------------+
| msg.delivery.v1
v
Gateway --> clients (via Redis pub/sub + WS)
Redis sits alongside the gateway and processors for presence reference-counting, typing/user-event fan-out, and async-send ack wake-ups. PostgreSQL is the source of truth for messages and delivery receipts; Kafka and Redis are best-effort distribution paths around that authoritative state. A fuller description of components, data flow, the concurrency model, failure assumptions, and trade-offs is in docs/architecture.md.
Prerequisites: Docker Desktop running, Git, and a shell (PowerShell on Windows, or any POSIX shell).
A system Go install is optional - a pinned toolchain is bundled at ohmf/.tools/go/bin/go.exe.
Bring up the local stack from the repo root:
docker compose up -d --build
docker compose psHealth-check the gateway (it is not published to the host by default):
docker compose exec gateway wget -qO- http://localhost:8081/healthzRun the Go tests for the gateway:
cd ohmf/services/gateway
../../.tools/go/bin/go.exe test ./... # Windows bundled toolchain
# or, with a system Go: go test ./...Run the repository test helper / containerized integration test:
./scripts/run-tests.sh --integration # POSIX
docker compose run --rm itest # containerized integration testsA larger compose stack that includes Kafka, Cassandra, Redis, and all processors lives at
ohmf/infra/docker/docker-compose.yml; see Full stack
below. For the complete day-to-day local-hosting guide, see
Local development guide.
Honest status: the repository does not contain a WebSocket concurrency harness, but it does
contain committed local benchmark artifacts under benchmarks/results/.
M4 validation summary — each row links to the committed artifact:
| Scenario | Result | Artifact |
|---|---|---|
Normal scaled load: configured 120 msg/sec, 4 processors, 12 source IPs |
Historical count-based settlement — 74,700/74,700 accepted, Postgres count 74,700, Cassandra count 74,700, Kafka lag→0 in 103s |
processor-scaling-matrix |
Pod deletion/rebalance: one processor pod deleted mid-run at configured 120 msg/sec |
Historical count-based settlement — 82,800/82,800 accepted, matching store counts, Kafka lag→0 in 283.608s; current recovery fixes not rerun |
processor-pod-deletion-120msgsec |
Backlog recovery: processors scaled to 0, peak lag 18,021, restored to 4 replicas |
Historical count-based settlement — 82,800/82,800 accepted, matching store counts, lag→0; current recovery fixes not rerun |
processor-backlog-recovery-120msgsec |
Diagnostic history: the initial 2026-06-09 pod-deletion run at 120 msg/sec was invalidated by a coincident Redis ack outage (4,520 gateway 500s); that artifact is preserved at 2026-06-09-processor-pod-deletion-120msgsec as diagnostic-only evidence of the Redis ack issue.
Current bounded evidence (2026-09-11): the remediated local K3d ladder passed exact
identity/content/order, Postgres, Cassandra, durable-side-effect, and Kafka-lag gates at
30/60/90/105/120 msg/sec for 30 seconds. High-concurrency 180/240/300/330 msg/sec cells and a
max-concurrency 360 msg/sec cell passed the same gates; the 360 cell reconciled 10,800/10,800
with schedule-delay p95 0.625 ms and 336.715 valid durable msg/sec. A 640-worker 360 control
and the earlier 40-worker 180 cell reconciled exactly but failed pacing validity, so they remain
diagnostics. The current bound is local and bounded, not a production or unbounded capacity claim.
See benchmarks/RESULTS_INDEX.md for authoritative, valid,
diagnostic, and quarantined classifications.
Path to these results: Stage A smoke at 5 msg/sec → per-user and per-IP limiter validations → Stage
B1 historical rerun ladder reporting matching store counts at 75, 90, and 105 msg/sec (12 source IPs, 1
processor). The 105 msg/sec result remains the previous single-processor-supported rung. Earlier
failures (60 msg/sec under-reconciliation traced to stale image; 120 msg/sec single-processor lag
not settling) are preserved in the artifact history as diagnostic context.
The historical artifacts above do not substantiate large-client-count or production-throughput
claims. The current isolated timing cell reports corrected client E2E send-to-response latency
(p50/p95/p99 76.084/330.742/755.190 ms) plus processor receive/fetch and fetch/handler-complete
histograms; those backend histograms are not joined per message. The old
ohmf/services/gateway/_tools/e2ee-load-test.go remains
an in-process simulation of E2EE message generation — it does not open WebSocket connections,
does not measure end-to-end message loss. Treat it as a micro-benchmark scaffold, not as system
throughput evidence.
Benchmark documentation is under benchmarks/, which describes what a credible run must capture so results can be reproduced rather than asserted.
.
|-- docker-compose.yml # root local stack (Postgres + gateway + integration tests)
|-- docs/
| |-- architecture.md # components, data flow, concurrency, trade-offs, limits
| `-- reliability-hardening.md# correctness case study, each fix linked to its test
|-- benchmarks/ # load-test methodology + status (see README)
|-- scripts/ # cross-platform test runners
|-- ohmf/
| |-- services/
| | |-- gateway/ # HTTP + WebSocket API, realtime, messages, e2ee, presence
| | |-- messages-processor/ # Kafka consumer: persist + retry semantics
| | |-- delivery-processor/ # delivery receipts, idempotent
| | |-- sms-processor/ # SMS dispatch
| | `-- contacts | apps | media | auth | users | ...
| |-- infra/
| | |-- docker/ # full stack: Kafka, Cassandra, Redis, processors
| | `-- observability/ # Prometheus + Grafana
| |-- apps/ # web client + Android scaffold
| `-- packages/protocol/ # OpenAPI + SQL schema
`-- .github/workflows/ # CI: OpenAPI validation, Go tests, docker build, integration
These are stated up front so the repo is read accurately:
-
Delivery semantics are at-least-once, not exactly-once. Postgres is authoritative; Kafka and Redis are best-effort around it. A crash between a side effect and its idempotency marker can duplicate that side effect on retry. (See
docs/reliability-hardening.mdsection 5.) -
Kubernetes is local-single-node only.
deploy/k8s/now contains:- a stage-1 smoke profile (
local-k3s) for gateway-smoke +apps+ Postgres/Redis - a fuller local pipeline profile (
local-k3s-full) that deploys Kafka, Cassandra, the full gateway, andmessages-processor - a local HPA layer (
local-k3s-full-hpa) that was smoke-tested against the gateway
Recorded artifacts under
deploy/k8s/results/show these profiles working on a real single-node cluster: asyncgateway/API -> Kafka -> processor -> Postgres/Cassandraproof, gateway HPA smoke, and M4 load validations at 120 msg/sec (scaling matrix, pod deletion/rebalance, backlog recovery). These artifacts do not support production operations claims, Helm, multi-node resilience, durable storage, ingress/TLS, network policy, or HA failover claims. - a stage-1 smoke profile (
-
Cassandra is in shadow-write mode. It is wired up and written to, but reads default to Postgres (
APP_USE_CASSANDRA_READS=false). The Cassandra read path is not the live serving path. -
No substantiated production load-test results. See Benchmarks.
-
Ordering is per-conversation, not global. Fan-out preserves
server_orderwithin a conversation, not a total order across conversations.
The sections below are the detailed local-hosting reference. The Quickstart above is enough to get the stack running; this is for day-to-day work.
You do not manually populate postgres-data/. It is populated automatically by the postgres
container on first startup because the root docker-compose.yml bind-mounts it:
volumes:
- ./postgres-data:/var/lib/postgresql/data- First
docker compose up: Postgres initializes data files inpostgres-data/. - Next runs: existing data is reused (persistent local state).
- Hard reset:
docker compose down -v.
Inside the Docker network (service-to-service):
- Contacts:
http://contacts:18085 - Apps:
http://apps:18086 - Media:
http://media:18087 - Gateway:
http://gateway:8081
From your host machine:
- Postgres:
localhost:5432
To expose the gateway on your host, add to the gateway service in docker-compose.yml:
ports:
- "8081:8081"Then recreate it: docker compose up -d --build gateway. Host access is then at
http://localhost:8081/healthz.
# Windows, bundled toolchain
Push-Location .\ohmf\services\gateway
& ..\..\.tools\go\bin\go.exe test ./...
Pop-Location
# Repository helpers
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\run-tests.ps1 -Integrationchmod +x scripts/*.sh
./scripts/run-tests.sh --integration