English | 简体中文
Pulsar Lite is a lightweight local broker that implements the core Apache Pulsar binary protocol for development, integration testing, and small local prototypes.
It is designed for fast local feedback, not as a production replacement for an Apache Pulsar cluster. Use Apache Pulsar for production workloads that require multi-broker scheduling, replication, capacity management, tenant isolation, or operational SLAs.
Many applications only need a local broker to validate the messaging path: producers, consumers, subscriptions, flow control, failover, and key-based routing. Setting up a full Pulsar deployment can be more expensive than the test or prototype itself.
Pulsar Lite keeps the client-facing API close to Pulsar while reducing the local setup cost:
- Run a single local broker process.
- Connect with the official Pulsar Python client.
- Use Pulsar topic names such as
persistent://...andnon-persistent://.... - Exercise Shared, Failover, Exclusive, and KeyShared subscription behavior.
- Use RocksDB-backed persistent storage when built with
rocksdb-storage.
| Area | Status |
|---|---|
| Binary protocol | Core commands are implemented: Connect, Lookup, PartitionMetadata, Producer, Send, Subscribe, Flow, Ack, Close, Ping/Pong, redelivery paths. |
| Official client compatibility | The official Pulsar Python client can connect to pulsar://localhost:6650. |
| Topic names | Supports persistent://... and non-persistent://... topic URIs. |
| Non-persistent topics | Dispatch-or-drop runtime semantics with coverage for flow control, disconnect/reconnect, ordering, dynamic consumers, and KeyShared routing. |
| Persistent topics | RocksDB-backed managed-ledger style storage is available behind the rocksdb-storage feature. |
| Subscription modes | Shared, Failover, Exclusive, and KeyShared are covered by Rust and Python integration tests. |
| Monitoring | Prometheus metrics on GET /metrics (port 8080): broker/topic/subscription counters, backlog and unacked gauges, storage latency and entry-size histograms, process metrics. Optional Grafana stack under grafana/. |
| Partitioned topics | Default partition metadata and partition topic routing are supported for local testing. |
| Python package | Provides a small helper SDK that can start and manage a local broker process. |
- Rust stable with
rustfmtandclippy. - Python 3.10 or newer for the tested development workflow.
protobuf-compiler/protoc.- RocksDB build dependencies for
rocksdb-storage.
On Ubuntu:
sudo apt-get update
sudo apt-get install -y protobuf-compiler clang libclang-devOn macOS:
brew install protobuf llvmBuild the broker with persistent storage support:
cd rust
cargo build --release --features rocksdb-storageInstall the Python package in editable mode:
cd ../python
pip install -e ".[dev]"Start the local broker:
../rust/pulsar-lite.sh startConnect with the official Pulsar client:
import pulsar
client = pulsar.Client("pulsar://localhost:6650")
topic = "non-persistent://public/default/events"
consumer = client.subscribe(topic, "demo-sub", consumer_type=pulsar.ConsumerType.Shared)
producer = client.create_producer(topic)
producer.send(b"event-1")
message = consumer.receive(timeout_millis=5000)
consumer.acknowledge(message)
producer.close()
consumer.close()
client.close()Stop the broker:
../rust/pulsar-lite.sh stopThe broker exposes Prometheus metrics on GET /metrics (default
0.0.0.0:8080, same port model as native Pulsar's web service). Metric
families reuse native Pulsar names and labels (pulsar_rate_in,
pulsar_subscription_back_log, pulsar_storage_write_latency, ...), so
existing dashboards and PromQL translate directly; extensions unique to
Pulsar Lite use the pulsar_lite_* prefix.
Quick check:
curl -s localhost:8080/metrics | grep -E '^pulsar_(broker|subscription)'Configuration (rust/pulsar-lite.toml):
[metrics]
enabled = true # false: no listener, no scrape aggregation
addr = "0.0.0.0:8080"
cluster = "pulsar-lite" # `cluster` label value on every family
rate_window_secs = 60 # window for pulsar_rate_in-style gaugesA ready-to-run Prometheus + Grafana stack (with provisioned dashboards for
topics and broker overview) lives under grafana/:
docker compose -f grafana/docker-compose.yml up -d
# Grafana http://localhost:3000 (admin/admin), Prometheus http://localhost:9090The Python helper can start a local broker for short-lived tests or examples:
import pulsar
from pulsar_lite import PulsarClient
topic = "non-persistent://public/default/quick-start"
with PulsarClient("./demo.db") as client:
consumer = client.subscribe(topic, "quick-start-sub", consumer_type=pulsar.ConsumerType.Shared)
producer = client.create_producer(topic)
producer.send(b"hello from pulsar lite")
message = consumer.receive(timeout_millis=5000)
consumer.acknowledge(message)Pulsar Lite accepts standard Pulsar topic names:
persistent://public/default/my-topic
non-persistent://public/default/my-topic
Use non-persistent://... for live event dispatch where slow or disconnected
consumers should not create a durable backlog. Use persistent://... when a
test requires stored entries, cursor replay, acknowledgements across restart,
or redelivery behavior. Persistent behavior requires a broker binary built with
--features rocksdb-storage.
Supported subscription modes:
| Mode | Summary |
|---|---|
| Exclusive | One active consumer; additional consumers are rejected. |
| Failover | One active consumer with standby takeover. |
| Shared | Messages are distributed across available consumers. |
| KeyShared | Messages with the same key are routed to the same consumer. |
make build # Build the Rust broker with rocksdb-storage
make install # Install the Python package in editable mode
make test # Run Rust and Python tests
make test-rust # Run Rust tests with rocksdb-storage
make test-python # Run Python integration tests with a local broker
make fmt # Format Rust and Python code
make lint # Run Rust clippy and Python ruff checksThe Python integration suite expects a broker binary with RocksDB support:
cd rust
cargo build --release --features rocksdb-storage
cd ../python
PULSAR_LITE_BINARY=../rust/target/release/pulsar-lite pytest ../tests/ -qpulsar-lite/
├── rust/ # Rust broker implementation
│ ├── src/broker/ # Broker service, connection handling, dispatchers
│ ├── src/protocol/ # Pulsar binary protocol codec and commands
│ ├── src/storage/ # Metadata, resources, managed ledger, RocksDB storage
│ └── proto/ # Pulsar protobuf definitions
├── python/ # Python helper package and broker process manager
├── tests/ # Python integration and behavior tests
├── examples/ # Small Python usage examples
└── docs/ # Protocol, design, comparison, test, and perf notes
Pulsar Lite intentionally does not provide:
- Multi-broker coordination or load balancing.
- Cross-cluster replication.
- Production-grade authorization or tenant governance.
- BookKeeper compatibility.
- Production durability or availability guarantees.
The project is useful for local development and compatibility testing, but production deployments should use Apache Pulsar.
Pulsar Lite is licensed under the Apache License 2.0.