Site: https://bsommers.github.io/PostOffice/
PostOffice is a high-throughput, hybrid Rust/Python multi-protocol messaging system and semantic translation bridge routing data between MQTT, AMQP (RabbitMQ), Apache Kafka, and NanoMQ edge brokers with verified delivery guarantees.
-
Hybrid Native Rust + Python Architecture: High-speed Rust Data Plane core (
libpostofficeC ABI +NativeEngine) achieving 24x–253x micro-benchmark speedups and sub-millisecond p99 bridge latency, paired with an ergonomic PythonPostOfficefacade, Control Plane, and reference conformance oracle (71/71vectors in--bothmode). - Multi-Protocol Bridging: Seamless routing across MQTT, NanoMQ, RabbitMQ (AMQP), and Apache Kafka.
-
$O(k)$ TopicTrie Routing: Hierarchical topic matching with+(single-level) and#(multi-level) wildcards and MQTT$-prefix isolation (§5.0), scaling over 648x faster than linear scans. -
Ingress Firewall & DoS Protection: Pre-routing payload size caps, topic shape constraints, token-bucket rate limits, streaming JSON/Avro/Protobuf
ContentGuard,SenderLedgerwith escalating cool-downs,InFlightGateconcurrency limits,RetryLimiterpoison backoff, and cgroup-awareMemoryGovernor. -
End-to-End Delivery Guarantees: Asynchronous publisher confirmation coordination (
_FanoutCoordinator), contiguous Kafka offset watermarks, AMQP publisher confirms, MQTT QoS 1PUBACKtracking, and automatic semantic translation. -
Dynamic Schema Registry & Transformation: Runtime bidirectional payload conversion between JSON, Apache Avro (
fastavro), and Protocol Buffers (protobuf) with Confluent 5-byte wire format framing and schema complexity caps. -
Production Security & Hot-Reload: TLS/mTLS encryption, strict certificate verification, SASL authentication (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512), HMAC-SHA256 signed Control Plane events, and zero-downtime in-flight credential rotation (
reload_broker_security). -
Operational Resiliency & Edge Store-and-Forward: Offline subscription replay, per-target bounded publish queues (
max_messages,max_bytes,max_age_sec), opt-in edge disk-backed store-and-forward (DiskStoreAndForwardwith wear-aware batching and CRC32 crash recovery), and egressCircuitBreakerprotection. -
Mixed-Fleet Canary Rollout & Observability: Route-level engine tagging (
engine="python" | "native"), automatedCanaryRolloutController(5% -> 25% -> 100%with SLO auto-rollback), shadow mode differential testing, Prometheus/metrics,/healthz,/readyzprobes, structured JSON logs, and budgeted Dead Letter Queues. -
Distributed Control & Data Plane: Race-free subscribe-then-snapshot Redis synchronization with Pub/Sub (
postoffice:config_updates) and durable Redis Streams (postoffice:config_stream) replay.
from postoffice.app import PostOffice
# Initialize PostOffice (enable Native Rust Engine with use_native_engine=True)
po = PostOffice(
metrics_port=9100,
dlq_broker="kafka_bus",
dlq_topic="deadletter.events",
use_native_engine=True,
)
# Register broker connections
po.add_broker("mqtt_edge", "mqtt", host="localhost", port=1883)
po.add_broker("kafka_bus", "kafka", bootstrap_servers="localhost:9092")
# Add a wildcard route with dynamic schema conversion
po.add_route(
source_broker="mqtt_edge",
source_topic="sensors/+/telemetry",
target_broker="kafka_bus",
target_topic="analytics.readings",
input_format="json",
output_format="avro",
output_schema="SensorReading",
wire_format=True,
)
# Start connections
po.start()Comprehensive documentation, tutorials, contracts, and operational runbooks:
- 📖 User Guide Entrypoint — Installation, quickstart, and full API facade reference.
- 🔀 Topic Routing & Wildcards — MQTT
+/#wildcards,$-prefix isolation, and$O(k)$ TopicTrie benchmarks. - 🛡️ Delivery Guarantees & Semantic Translation — QoS 1/2 coordination, contiguous Kafka offsets, and persistent delivery modes.
- 🧱 Ingress Firewall, Sender Ledger & In-Flight Limits — Pre-routing DoS rules, content guard, sender cool-down, and retry caps.
- 🧬 Dynamic Schema Registry & Serialization — JSON, Avro, Protobuf, and Confluent wire framing.
- 🔒 Production Security, TLS/mTLS & Authentication — TLS/mTLS, SASL, unified
SecurityConfig, and live credential hot-reload. - 🛟 Operational Resiliency, Queuing & Circuit Breakers — Offline subscription replay, bounded publish queues, edge store-and-forward, and circuit breakers.
- 📊 Observability & Dead Letter Queues — Prometheus metrics,
/healthz&/readyzprobes, structured JSON logs, and budgeted DLQ quarantine. - 🌐 Distributed Control & Data Plane — Multi-worker scaling via Redis Pub/Sub & Streams, HMAC event signing, and canary rollouts.
- 📜 Polyglot v1 Contracts & Conformance — Protobuf IDL (
proto/postoffice/v1/postoffice.proto), OpenAPI 3.1 (api/openapi.v1.yaml), C ABI (include/postoffice.h), and 71-vector conformance suite. - 🚀 Mixed-Fleet Canary Rollout Runbook — Helm/Compose deployment, canary ramp (
5% -> 25% -> 100%), and emergency rollback procedures. - ⚡ Benchmarks & Native Bake-Off — Micro-benchmark comparisons, bridge-added latency histograms, and memory budgets.
- 📐 Architecture Decision Records (ADRs) — ADR-0001 (Rust Data Plane) and ADR-0002 (Phase 5 Consolidation).
- 💡 Use Cases & Purpose — How PostOffice fits in IIoT, IoT/edge, data systems, and enterprise messaging.
- 🔧 Troubleshooting & FAQ — Common error diagnostics, recovery steps, and solutions.
- 🏛️ Architecture & Protocol Mapping — Deep architectural overview and semantic translation tables.
- 🧪 Test Suite & Coverage Report — Module coverage breakdown (
92%combined branch + statement coverage across586Python tests +7Rust tests +71conformance vectors).
Execute the hermetic Python unit test suite (586 tests, no external broker services required):
PYTHONPATH=src .venv/bin/python -m unittest discover tests/Execute the dual-engine conformance suite (71/71 vectors verified across both python and native Rust engines):
PYTHONPATH=src .venv/bin/python conformance/run.py --bothExecute the native Rust unit and fuzz test suite:
cargo test --release --manifest-path native/libpostoffice/Cargo.tomlRun the coverage gate (enforces >= 90% combined statement + branch coverage via .coveragerc):
PYTHONPATH=src .venv/bin/coverage run -m unittest discover tests/ && .venv/bin/coverage report