Skip to content
f8573Public

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

201 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OHMF — Distributed Real-Time Messaging Backend

OHMF is a distributed real-time messaging backend written in Go. It accepts messages over an HTTP/WebSocket gateway, moves them through a Kafka event pipeline to background processors, and persists conversation state durably in PostgreSQL (with Cassandra wired in as a secondary message store). Redis carries presence and low-latency realtime fan-out. The whole stack runs locally under Docker Compose.

The project exists to be a concrete, inspectable example of the correctness problems that show up in a multi-service messaging system - idempotency under concurrent sends, per-conversation ordering across multiple workers, at-least-once delivery with duplicate suppression, and cross-pod presence - and to show those problems being solved with tests that pin the behavior down. It is a working development environment and a correctness case study, not a finished, production-operated system.

Historical local experiments at a configured 120 msg/sec reported matching accepted/Postgres/Cassandra counts and drained Kafka lag — all on a single-node local cluster, not production. Those artifacts predate current identity/content reconciliation and crash-window fixes. Core reliability hardening is implemented and regression-tested; a fresh benchmark and live fault-injection campaign remain pending. See Limitations and Benchmarks.

What this demonstrates

  • End-to-end backend ownership — gateway, Kafka pipeline, message processors, persistence, and delivery in a single coherent system
  • Distributed messaging pipeline design — at-least-once delivery with duplicate suppression, per-conversation ordering, and idempotency under concurrent retries
  • Real-time WebSocket / API behavior — presence tracking, typing indicators, fan-out across pods via Redis pub/sub
  • Historical load/failure evidence — named local artifacts cover scaled load, processor pod deletion/rebalance, and backlog recovery, but are count-based and do not validate the remediated crash windows
  • Operational docs and reproducible workflows — local Docker Compose and Kubernetes (k3s/k3d) bring-up, CI pipeline, and documented limitations

Benchmark evidence

Full evidence chain for the M4 validation milestone:

Document Contents
docs/claims-and-evidence.md Per-claim status, artifact links, and supported claim boundary
docs/validation-matrix.md Test-by-test pass/fail table across all stages
benchmarks/results/2026-06-09-processor-scaling-matrix/summary.json Historical scaled load: matching counts and drained lag
benchmarks/results/2026-06-10-processor-pod-deletion-120msgsec/summary.json Historical pod deletion/rebalance: count-based settlement; current fixes not rerun
benchmarks/results/2026-06-10-processor-backlog-recovery-120msgsec/summary.md Historical backlog recovery: count-based settlement after lag drain

All results are local single-node cluster only. See Limitations for the full boundary.

Why this exists

Most "chat app" projects stop at create-read-update-delete over a database. The interesting and hard part of messaging is everything around that: what happens when the same send is retried, when two workers fan out the same conversation concurrently, when a delivery event is redelivered, or when a user is connected to two pods at once. OHMF is built to make those failure paths explicit and to demonstrate the engineering - not just the happy path. The reliability work is documented with the specific test that proves each behavior in docs/reliability-hardening.md.

Recent hardening also covers privacy-sensitive paths that are easy to miss in ordinary feature delivery: reaction history/export filtering, typing.stop privacy handling, websocket heartbeat cleanup, cross-pod presence teardown, retryable fanout on Redis publish failure, persistent worker failure surfacing, and per-conversation fanout ordering.

For a compact evidence trail, see docs/claims-and-evidence.md and the committed benchmark artifacts under benchmarks/results/. Those records show reliability and privacy hardening as first-class system behavior, not invisible maintenance.

Stack

What is actually present and used in the code (verified against go.mod files and the compose stack):

Layer Technology Where
Language Go 1.25 (multi-module) ohmf/services/*
API / realtime HTTP + WebSocket gateway (go-chi/chi, gorilla/websocket) ohmf/services/gateway
Event pipeline Apache Kafka (segmentio/kafka-go) ohmf/services/gateway/internal/bus, processors
Authoritative store PostgreSQL gateway + processors
Secondary message store Cassandra (gocql) - shadow-write enabled, reads off by default ohmf/services/gateway/internal/messages/cassandra_store.go
Presence / realtime fan-out Redis gateway, processors
Observability Prometheus + Grafana dashboards ohmf/infra/observability
Local orchestration Docker Compose docker-compose.yml, ohmf/infra/docker
Web client Static HTML/CSS/vanilla JS ohmf/apps/web
CI GitHub Actions (OpenAPI validation, Go tests, docker build, containerized integration test) .github/workflows

Kubernetes: plain-Kubernetes (Kustomize, no Helm) manifests for local single-node k3s/k3d validation live under deploy/k8s/. The repo now contains a lighter smoke profile (overlays/local-k3s), a fuller local pipeline profile (overlays/local-k3s-full), and an optional local gateway HPA layer (overlays/local-k3s-full-hpa). These support local single-node evidence only: they are not a production deployment package, not multi-node resilience evidence, not Helm, and not benchmark-validated. See deploy/k8s/README.md.

Not currently in the repo: Helm charts and standalone WebSocket load-test client scripts (the benchmark load-generators run as Kubernetes Jobs, not standalone scripts). These are referenced as design intent only and are called out below.

Architecture

                       WebSocket / HTTP clients
                                 |
                                 v
                      +----------------------+
                      |   Gateway (API)      |  auth, validation, idempotency,
                      |  chi + gorilla/ws    |  realtime fan-out
                      +----------------------+
                                 | produce: msg.ingress.v1
                                 v
                          +-------------+
                          |    Kafka    |  partitioned topics + DLQs
                          +-------------+
                                 | consume
                                 v
                +-----------------------------------+
                | messages-processor                |  persist + retry semantics
                +-----------------------------------+
                     |                 |
                     v                 v
              +------------+    +------------+
              | PostgreSQL |    | Cassandra  |  (authoritative)   (shadow write)
              +------------+    +------------+
                     | produce: msg.persisted.v1
                     v
                +-----------------------------------+
                | delivery-processor / sms-processor|  delivery receipts, SMS dispatch
                +-----------------------------------+
                                 | msg.delivery.v1
                                 v
                        Gateway --> clients (via Redis pub/sub + WS)

Redis sits alongside the gateway and processors for presence reference-counting, typing/user-event fan-out, and async-send ack wake-ups. PostgreSQL is the source of truth for messages and delivery receipts; Kafka and Redis are best-effort distribution paths around that authoritative state. A fuller description of components, data flow, the concurrency model, failure assumptions, and trade-offs is in docs/architecture.md.

Quickstart

Prerequisites: Docker Desktop running, Git, and a shell (PowerShell on Windows, or any POSIX shell). A system Go install is optional - a pinned toolchain is bundled at ohmf/.tools/go/bin/go.exe.

Bring up the local stack from the repo root:

docker compose up -d --build
docker compose ps

Health-check the gateway (it is not published to the host by default):

docker compose exec gateway wget -qO- http://localhost:8081/healthz

Run the Go tests for the gateway:

cd ohmf/services/gateway
../../.tools/go/bin/go.exe test ./...      # Windows bundled toolchain
# or, with a system Go: go test ./...

Run the repository test helper / containerized integration test:

./scripts/run-tests.sh --integration            # POSIX
docker compose run --rm itest                    # containerized integration tests

A larger compose stack that includes Kafka, Cassandra, Redis, and all processors lives at ohmf/infra/docker/docker-compose.yml; see Full stack below. For the complete day-to-day local-hosting guide, see Local development guide.

Benchmarks and load testing

Honest status: the repository does not contain a WebSocket concurrency harness, but it does contain committed local benchmark artifacts under benchmarks/results/.

M4 validation summary — each row links to the committed artifact:

Scenario Result Artifact
Normal scaled load: configured 120 msg/sec, 4 processors, 12 source IPs Historical count-based settlement — 74,700/74,700 accepted, Postgres count 74,700, Cassandra count 74,700, Kafka lag→0 in 103s processor-scaling-matrix
Pod deletion/rebalance: one processor pod deleted mid-run at configured 120 msg/sec Historical count-based settlement — 82,800/82,800 accepted, matching store counts, Kafka lag→0 in 283.608s; current recovery fixes not rerun processor-pod-deletion-120msgsec
Backlog recovery: processors scaled to 0, peak lag 18,021, restored to 4 replicas Historical count-based settlement — 82,800/82,800 accepted, matching store counts, lag→0; current recovery fixes not rerun processor-backlog-recovery-120msgsec

Diagnostic history: the initial 2026-06-09 pod-deletion run at 120 msg/sec was invalidated by a coincident Redis ack outage (4,520 gateway 500s); that artifact is preserved at 2026-06-09-processor-pod-deletion-120msgsec as diagnostic-only evidence of the Redis ack issue.

Current bounded evidence (2026-09-11): the remediated local K3d ladder passed exact identity/content/order, Postgres, Cassandra, durable-side-effect, and Kafka-lag gates at 30/60/90/105/120 msg/sec for 30 seconds. High-concurrency 180/240/300/330 msg/sec cells and a max-concurrency 360 msg/sec cell passed the same gates; the 360 cell reconciled 10,800/10,800 with schedule-delay p95 0.625 ms and 336.715 valid durable msg/sec. A 640-worker 360 control and the earlier 40-worker 180 cell reconciled exactly but failed pacing validity, so they remain diagnostics. The current bound is local and bounded, not a production or unbounded capacity claim. See benchmarks/RESULTS_INDEX.md for authoritative, valid, diagnostic, and quarantined classifications.

Path to these results: Stage A smoke at 5 msg/sec → per-user and per-IP limiter validations → Stage B1 historical rerun ladder reporting matching store counts at 75, 90, and 105 msg/sec (12 source IPs, 1 processor). The 105 msg/sec result remains the previous single-processor-supported rung. Earlier failures (60 msg/sec under-reconciliation traced to stale image; 120 msg/sec single-processor lag not settling) are preserved in the artifact history as diagnostic context.

The historical artifacts above do not substantiate large-client-count or production-throughput claims. The current isolated timing cell reports corrected client E2E send-to-response latency (p50/p95/p99 76.084/330.742/755.190 ms) plus processor receive/fetch and fetch/handler-complete histograms; those backend histograms are not joined per message. The old ohmf/services/gateway/_tools/e2ee-load-test.go remains an in-process simulation of E2EE message generation — it does not open WebSocket connections, does not measure end-to-end message loss. Treat it as a micro-benchmark scaffold, not as system throughput evidence.

Benchmark documentation is under benchmarks/, which describes what a credible run must capture so results can be reproduced rather than asserted.

Project structure

.
|-- docker-compose.yml          # root local stack (Postgres + gateway + integration tests)
|-- docs/
|   |-- architecture.md         # components, data flow, concurrency, trade-offs, limits
|   `-- reliability-hardening.md# correctness case study, each fix linked to its test
|-- benchmarks/                 # load-test methodology + status (see README)
|-- scripts/                    # cross-platform test runners
|-- ohmf/
|   |-- services/
|   |   |-- gateway/            # HTTP + WebSocket API, realtime, messages, e2ee, presence
|   |   |-- messages-processor/ # Kafka consumer: persist + retry semantics
|   |   |-- delivery-processor/ # delivery receipts, idempotent
|   |   |-- sms-processor/      # SMS dispatch
|   |   `-- contacts | apps | media | auth | users | ...
|   |-- infra/
|   |   |-- docker/             # full stack: Kafka, Cassandra, Redis, processors
|   |   `-- observability/      # Prometheus + Grafana
|   |-- apps/                   # web client + Android scaffold
|   `-- packages/protocol/      # OpenAPI + SQL schema
`-- .github/workflows/          # CI: OpenAPI validation, Go tests, docker build, integration

Limitations

These are stated up front so the repo is read accurately:

  • Delivery semantics are at-least-once, not exactly-once. Postgres is authoritative; Kafka and Redis are best-effort around it. A crash between a side effect and its idempotency marker can duplicate that side effect on retry. (See docs/reliability-hardening.md section 5.)

  • Kubernetes is local-single-node only. deploy/k8s/ now contains:

    • a stage-1 smoke profile (local-k3s) for gateway-smoke + apps + Postgres/Redis
    • a fuller local pipeline profile (local-k3s-full) that deploys Kafka, Cassandra, the full gateway, and messages-processor
    • a local HPA layer (local-k3s-full-hpa) that was smoke-tested against the gateway

    Recorded artifacts under deploy/k8s/results/ show these profiles working on a real single-node cluster: async gateway/API -> Kafka -> processor -> Postgres/Cassandra proof, gateway HPA smoke, and M4 load validations at 120 msg/sec (scaling matrix, pod deletion/rebalance, backlog recovery). These artifacts do not support production operations claims, Helm, multi-node resilience, durable storage, ingress/TLS, network policy, or HA failover claims.

  • Cassandra is in shadow-write mode. It is wired up and written to, but reads default to Postgres (APP_USE_CASSANDRA_READS=false). The Cassandra read path is not the live serving path.

  • No substantiated production load-test results. See Benchmarks.

  • Ordering is per-conversation, not global. Fan-out preserves server_order within a conversation, not a total order across conversations.


Local development guide

The sections below are the detailed local-hosting reference. The Quickstart above is enough to get the stack running; this is for day-to-day work.

About postgres-data (root stack)

You do not manually populate postgres-data/. It is populated automatically by the postgres container on first startup because the root docker-compose.yml bind-mounts it:

volumes:
  - ./postgres-data:/var/lib/postgresql/data
  • First docker compose up: Postgres initializes data files in postgres-data/.
  • Next runs: existing data is reused (persistent local state).
  • Hard reset: docker compose down -v.

Local service endpoints

Inside the Docker network (service-to-service):

  • Contacts: http://contacts:18085
  • Apps: http://apps:18086
  • Media: http://media:18087
  • Gateway: http://gateway:8081

From your host machine:

  • Postgres: localhost:5432

To expose the gateway on your host, add to the gateway service in docker-compose.yml:

ports:
  - "8081:8081"

Then recreate it: docker compose up -d --build gateway. Host access is then at http://localhost:8081/healthz.

Running tests

# Windows, bundled toolchain
Push-Location .\ohmf\services\gateway
& ..\..\.tools\go\bin\go.exe test ./...
Pop-Location

# Repository helpers
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\run-tests.ps1 -Integration
chmod +x scripts/*.sh
./scripts/run-tests.sh --integration

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages