Skip to content

About

PostOffice - Multi-protocol messaging system using MQTT, AMQP, Kafka and other messaging systems

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

PostOffice

PostOffice

Site: https://bsommers.github.io/PostOffice/

PostOffice is a high-throughput, hybrid Rust/Python multi-protocol messaging system and semantic translation bridge routing data between MQTT, AMQP (RabbitMQ), Apache Kafka, and NanoMQ edge brokers with verified delivery guarantees.


Key Features

  • Hybrid Native Rust + Python Architecture: High-speed Rust Data Plane core (libpostoffice C ABI + NativeEngine) achieving 24x–253x micro-benchmark speedups and sub-millisecond p99 bridge latency, paired with an ergonomic Python PostOffice facade, Control Plane, and reference conformance oracle (71/71 vectors in --both mode).
  • Multi-Protocol Bridging: Seamless routing across MQTT, NanoMQ, RabbitMQ (AMQP), and Apache Kafka.
  • $O(k)$ TopicTrie Routing: Hierarchical topic matching with + (single-level) and # (multi-level) wildcards and MQTT $-prefix isolation (§5.0), scaling over 648x faster than linear scans.
  • Ingress Firewall & DoS Protection: Pre-routing payload size caps, topic shape constraints, token-bucket rate limits, streaming JSON/Avro/Protobuf ContentGuard, SenderLedger with escalating cool-downs, InFlightGate concurrency limits, RetryLimiter poison backoff, and cgroup-aware MemoryGovernor.
  • End-to-End Delivery Guarantees: Asynchronous publisher confirmation coordination (_FanoutCoordinator), contiguous Kafka offset watermarks, AMQP publisher confirms, MQTT QoS 1 PUBACK tracking, and automatic semantic translation.
  • Dynamic Schema Registry & Transformation: Runtime bidirectional payload conversion between JSON, Apache Avro (fastavro), and Protocol Buffers (protobuf) with Confluent 5-byte wire format framing and schema complexity caps.
  • Production Security & Hot-Reload: TLS/mTLS encryption, strict certificate verification, SASL authentication (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512), HMAC-SHA256 signed Control Plane events, and zero-downtime in-flight credential rotation (reload_broker_security).
  • Operational Resiliency & Edge Store-and-Forward: Offline subscription replay, per-target bounded publish queues (max_messages, max_bytes, max_age_sec), opt-in edge disk-backed store-and-forward (DiskStoreAndForward with wear-aware batching and CRC32 crash recovery), and egress CircuitBreaker protection.
  • Mixed-Fleet Canary Rollout & Observability: Route-level engine tagging (engine="python" | "native"), automated CanaryRolloutController (5% -> 25% -> 100% with SLO auto-rollback), shadow mode differential testing, Prometheus /metrics, /healthz, /readyz probes, structured JSON logs, and budgeted Dead Letter Queues.
  • Distributed Control & Data Plane: Race-free subscribe-then-snapshot Redis synchronization with Pub/Sub (postoffice:config_updates) and durable Redis Streams (postoffice:config_stream) replay.

Quickstart

from postoffice.app import PostOffice

# Initialize PostOffice (enable Native Rust Engine with use_native_engine=True)
po = PostOffice(
    metrics_port=9100,
    dlq_broker="kafka_bus",
    dlq_topic="deadletter.events",
    use_native_engine=True,
)

# Register broker connections
po.add_broker("mqtt_edge", "mqtt", host="localhost", port=1883)
po.add_broker("kafka_bus", "kafka", bootstrap_servers="localhost:9092")

# Add a wildcard route with dynamic schema conversion
po.add_route(
    source_broker="mqtt_edge",
    source_topic="sensors/+/telemetry",
    target_broker="kafka_bus",
    target_topic="analytics.readings",
    input_format="json",
    output_format="avro",
    output_schema="SensorReading",
    wire_format=True,
)

# Start connections
po.start()

Documentation

Comprehensive documentation, tutorials, contracts, and operational runbooks:


Running Tests, Conformance & Coverage Gates

Execute the hermetic Python unit test suite (586 tests, no external broker services required):

PYTHONPATH=src .venv/bin/python -m unittest discover tests/

Execute the dual-engine conformance suite (71/71 vectors verified across both python and native Rust engines):

PYTHONPATH=src .venv/bin/python conformance/run.py --both

Execute the native Rust unit and fuzz test suite:

cargo test --release --manifest-path native/libpostoffice/Cargo.toml

Run the coverage gate (enforces >= 90% combined statement + branch coverage via .coveragerc):

PYTHONPATH=src .venv/bin/coverage run -m unittest discover tests/ && .venv/bin/coverage report

About

PostOffice - Multi-protocol messaging system using MQTT, AMQP, Kafka and other messaging systems

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages