diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5f189a5..1fde91f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -32,12 +32,13 @@ jobs: run: >- pip install "radical.asyncflow==0.5.1" "radical.orbit==0.5.0" "rhapsody-py[telemetry] @ git+https://github.com/radical-cybertools/rhapsody@e491cd2" + "rose @ git+https://github.com/radical-cybertools/ROSE@64330d9cb43c3e13ca67daf0d8ae84a2ae6c3f17" # src/ layout: pytest tests the *installed* package, never the # working tree. Always install before testing -- a stale install # will silently pass against old code. - name: Install digitaltwin (test + service extras) - run: pip install ".[test,service]" pytest-timeout + run: pip install ".[test,service,learn]" pytest-timeout # --continue-on-collection-errors: one broken test module (see the # PR description) must not hide the rest of the suite behind an @@ -71,12 +72,13 @@ jobs: run: >- pip install "radical.asyncflow==0.5.1" "radical.orbit==0.5.0" "rhapsody-py[telemetry] @ git+https://github.com/radical-cybertools/rhapsody@e491cd2" + "rose @ git+https://github.com/radical-cybertools/ROSE@64330d9cb43c3e13ca67daf0d8ae84a2ae6c3f17" # src/ layout: pytest tests the *installed* package, never the # working tree. Always install before testing -- a stale install # will silently pass against old code. - name: Install digitaltwin (test + service extras) - run: pip install ".[test,service]" pytest-timeout + run: pip install ".[test,service,learn]" pytest-timeout # The ORBIT broker needs a self-signed TLS cert/key and a shared # ingress token at the default ~/.radical/orbit location -- see diff --git a/README.md b/README.md index 38f14b3..27618ec 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ Main set of features implemented: - Utility Tasks - Persistent Tasks - Callbacks -- Simple ZMQ pubsub backend +- Two pubsub backends: ZMQ, and ORBIT eventing - Graph builder - Convert to a Python Package - Several Tests / Examples @@ -69,7 +69,57 @@ are cloudpickled, so anyone who can reach the broker ports can execute code in every subscriber. A non-loopback bind needs an explicit configuration and a private/firewalled network. External channels are decoded with the codec their binding names: `json` (the default) and `raw` are safe to -accept from a producer you do not control, `cloudpickle` is not. +accept from a producer you do not control, `cloudpickle` is not. The +demos are the reason the ZMQ backend exists; anything beyond a laptop +should be on the ORBIT one below. + +## Choosing a data plane + +`DT_STREAM_BACKEND` picks which transport carries the twins' streams. It +is a **deployment-time** choice, resolved once where the framework runs; +no client and no session can ask for a different one. Nothing above +`PubSubBackend` -- not `DTRuntime`, not a component, not the injected +`RuntimeAPI.stream` client -- knows which is in use. + +| `DT_STREAM_BACKEND` | transport | ports it opens | use | +|---------------------|-----------|----------------|-----| +| `zmq` (default) | the framework's own XSUB/XPUB broker | two, unauthenticated, loopback by default | local, demos, the two-terminal loop | +| `orbit` | ORBIT eventing (`radical.orbit`) | **none** | anything shared, and everything in production | + +```sh +# a service deployment with the data plane inside the token domain +DT_STREAM_BACKEND=orbit radical-orbit-broker.py --plugins default,dt +``` + +An external subscriber joins the same way -- as an ORBIT participant, so +it needs the broker URL and the token, and no addresses at all: + +```python +from digitaltwin.streaming import connect_stream_client + +stream = await connect_stream_client(twin_id, backend='orbit') +await stream.subscribe_to_dtype(ECHO, queue) +``` + +**Payload ceiling**: an ORBIT frame is capped at 4 MiB, so a single +stream message must cloudpickle to less than that (64 KiB of the budget +is reserved for the envelope). Oversized payloads raise a clear +`ValueError` at `publish` -- ORBIT itself would drop the frame with +nothing but a log line, which on a days-long twin is indistinguishable +from a stalled stream. The ZMQ backend has no such ceiling; a twin meant +to run on either should stay well under it. Chunk large artifacts, or +stream a reference and move the bytes with the staging plugin. + +**Semantics** are the same on both: at-most-once, with bounded +drop-oldest queues (broker-side, and again on the hop into the host +loop). That *is* the DT conflation contract, so nothing above the +backend adds a second one -- a slow consumer loses samples rather than +memory, and never backpressures a producer. Loss is visible as a gap in +the broker-assigned sequence numbers. ORBIT's `replay` plugin would give +late joiners history; it is deliberately not integrated in v1. + +`perf/bench_streams.py` measures what the choice costs: about a +millisecond per stream hop on loopback. ## Running it as a service (the `dt` ORBIT plugin) @@ -195,17 +245,43 @@ again, so a `twin_create` after it fails immediately with `engine `ready` and stalling. `unregister_session`, then build the session and its twins again. -### Binding policy for the service (R7) +### The data plane and its trust boundary (R7) -The plugin runs its own DT stream broker, embedded, one per plugin and -shared by every twin. **It binds to loopback on a random port by -default, and that default is the safe one**: its payloads are -cloudpickled, so anyone who can reach the XSUB/XPUB ports gets code -execution in every subscriber -- weaker than the token-authenticated -ORBIT channel around it. +The DT streams carry cloudpickled payloads. That is accepted -- the +service already executes client-shipped component classes, and both sit +inside ORBIT's single-token trust domain (risk R4). What was *not* +acceptable is where those payloads used to travel: a pair of ZMQ ports +that authenticate nobody, so anyone who could reach them got code +execution in every subscriber, no token required. The data plane was +weaker than the control plane wrapped around it. + +**`DT_STREAM_BACKEND=orbit` closes that gap**, and a production +deployment must use it: + +```sh +DT_STREAM_BACKEND=orbit radical-orbit-broker.py --plugins default,dt +``` -A non-loopback bind is possible (`DT_STREAM_PUB_ADDR` / -`DT_STREAM_SUB_ADDR` on the service host) but requires a deliberate -decision *and* a firewalled or private network. Until the data plane -moves inside ORBIT's authenticated channel, do not expose those ports -- -including in demos. +The twins' streams become ORBIT events on the same token-authenticated +WebSocket star as every other call, under one `dt_stream` plugin +namespace. The embedded ZMQ broker is then **never started** -- the +service opens no data-plane port at all, and there is nothing left to +firewall. The payloads are still cloudpickle; what changed is that +reaching them now requires the same token as calling `twin_create`. +Reviewers can check the guarantee directly: the plugin host has no child +processes, and `admin/sessions` reports `{"stream_broker": {"backend": +"orbit"}}`. + +Two things this does *not* do. It does not make the payloads safe to +receive from an untrusted party -- per-tenant auth is post-v1, so +everything inside the token domain is still mutually trusting. And it +does not remove the 4 MiB frame cap, which the ZMQ backend did not have +(see "Choosing a data plane"). + +**With the `zmq` backend the old mitigations still apply, in full.** The +plugin runs its own DT stream broker, embedded, one per plugin and shared +by every twin. It binds to loopback on a random port by default, and +that default is the safe one. A non-loopback bind is possible +(`DT_STREAM_PUB_ADDR` / `DT_STREAM_SUB_ADDR` on the service host) but +requires a deliberate decision *and* a firewalled or private network. Do +not expose those ports -- including in demos. diff --git a/perf/README.md b/perf/README.md index 3469e77..3cd0931 100644 --- a/perf/README.md +++ b/perf/README.md @@ -29,6 +29,40 @@ Routing all user compute through the Rhapsody abstraction therefore costs single-digit milliseconds per sequential prediction and wins by an order of magnitude under concurrency. +## `bench_streams.py` — stream latency, ZMQ vs ORBIT data plane + +One publish awaited until the subscriber's queue hands it back: the shape +of every hop in a twin's graph. Both rows go through the same +`PubSubClient`, so the only difference is the backend (M3). + +```sh +# the zmq row starts its own embedded broker; the orbit row needs a live one +python perf/bench_streams.py both --broker https://127.0.0.1:8031 +``` + +Loopback, one host, bare-int payloads (2026-08-15): + +| data plane | p50 | p99 | burst | +|-----------------------------|---------|---------|----------------| +| zmq, embedded broker | 0.98 ms | 1.32 ms | 20 200 msg/s | +| orbit eventing | 2.09 ms | 2.60 ms | 4 300 msg/s | + +With 64 KiB payloads (`--payload 65536`): 1.18 ms / 3.56 ms p50, and +6 500 vs 1 150 msg/s in burst. + +So the ORBIT data plane costs roughly **1 ms per stream hop** and about a +quarter of the burst throughput, in exchange for the security property of +M3: the payloads ride the token-authenticated WebSocket star and the +deployment opens no unauthenticated ports (risk R7). Against the ~20 ms +of a single in-situ prediction (the row above), that is noise. +Informational, not a gate. + +The extra hop is structural: ZMQ's XSUB/XPUB proxy forwards a frame +between two sockets, while an ORBIT event is packed, sent to the broker, +stamped with a `seq`, fanned out, and handed across a thread boundary +into the host loop. Payload size hurts the ORBIT row more because the +frame is msgpacked around the pickle. + ## `streaming_learner_perf.py`, `plot_streaming_perf.py` Throughput of the streaming active learner (ROSE); unrelated to the diff --git a/perf/bench_streams.py b/perf/bench_streams.py new file mode 100644 index 0000000..3d98990 --- /dev/null +++ b/perf/bench_streams.py @@ -0,0 +1,177 @@ +"""Stream latency: the ZMQ data plane vs the ORBIT one. + +One publish, awaited until the subscriber's queue hands it back -- the +shape of every hop in a twin's graph (a persistent component publishes a +dtype, the runtime consumes it). Both rows go through the *same* +`PubSubClient`, so the difference is the backend and nothing else. + +The ZMQ row starts its own embedded broker on a random loopback port, +exactly as the service does. The ORBIT row needs a live broker:: + + # against the integration stack (test/integration/conftest.py has one) + python perf/bench_streams.py both --broker https://127.0.0.1:8031 + +Informational, not a gate: the ORBIT row buys the security property of +milestone M3 (no unauthenticated ports) and pays a WebSocket round trip +through the broker for it. +""" + +import argparse +import asyncio +import os +import sys +import time + +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from bench_insitu import report # noqa: E402 + +from digitaltwin import DataType, ZMQ_BrokerProcess # noqa: E402 +from digitaltwin.config import BACKEND_ORBIT, BACKEND_ZMQ # noqa: E402 +from digitaltwin.streaming import connect_stream_client # noqa: E402 + +SAMPLE = DataType("sample") + +N_WARM = 20 +N_MEAS = 200 +N_BURST = 200 + + +SETTLE_TIMEOUT = 30.0 +DELIVER_TIMEOUT = 10.0 + + +QUIET_WAIT = 0.25 + + +async def settle(client, queue, message) -> None: + """Publish until something comes back, then wait for quiet. + + Neither backend acknowledges a subscription -- ZMQ's SUBSCRIBE and + ORBIT's `subscribe` frame are both fire-and-forget -- so the only + honest barrier is a message that made the round trip. + + Draining until the queue *stays* empty matters as much as the barrier + itself: a `queue.empty()` check would leave any barrier message still + in flight to arrive during the measurement, where it would pair with + the wrong publish and shift every latency after it. + """ + + deadline = time.perf_counter() + SETTLE_TIMEOUT + + while time.perf_counter() < deadline: + await client.publish(SAMPLE, message) + try: + await asyncio.wait_for(queue.get(), QUIET_WAIT) + break + except TimeoutError: + continue + else: + raise TimeoutError(f"no message came back within {SETTLE_TIMEOUT}s") + + while time.perf_counter() < deadline: + try: + await asyncio.wait_for(queue.get(), QUIET_WAIT) + except TimeoutError: # nothing left in flight + return + + raise TimeoutError(f"the stream never went quiet within {SETTLE_TIMEOUT}s") + + +async def measure(client, label: str, n_meas: int, n_burst: int, + payload: int) -> None: + """Publish -> deliver round trips through one stream client.""" + + queue: asyncio.Queue = asyncio.Queue() + await client.subscribe_to_dtype(SAMPLE, queue) + + message = b"x" * payload if payload else 0 + + await settle(client, queue, message) + + for _ in range(N_WARM): + await client.publish(SAMPLE, message) + await asyncio.wait_for(queue.get(), DELIVER_TIMEOUT) + + latencies = [] + for _ in range(n_meas): + start = time.perf_counter() + await client.publish(SAMPLE, message) + await asyncio.wait_for(queue.get(), DELIVER_TIMEOUT) + latencies.append(time.perf_counter() - start) + + report(label, latencies) + + # burst: how fast the data plane drains a producer that does not wait. + # Both backends drop the oldest when a queue overruns, so the count + # that arrives is part of the measurement. + start = time.perf_counter() + for _ in range(n_burst): + await client.publish(SAMPLE, message) + + received = 0 + while received < n_burst: + try: + await asyncio.wait_for(queue.get(), 5.0) + except TimeoutError: + break + received += 1 + + elapsed = time.perf_counter() - start + print(f"{label:28s} {received}/{n_burst} burst messages in " + f"{elapsed * 1000:.1f}ms ({received / elapsed:.0f} msg/s)") + + +async def bench_zmq(args) -> None: + broker = ZMQ_BrokerProcess() + await broker.start() + + try: + client = await connect_stream_client( + "bench-zmq", *broker.get_connection_str(), backend=BACKEND_ZMQ + ) + try: + await measure(client, "zmq (embedded broker)", args.messages, + args.burst, args.payload) + finally: + await client.close() + finally: + await broker.stop() + + +async def bench_orbit(args) -> None: + client = await connect_stream_client( + "bench-orbit", backend=BACKEND_ORBIT, broker_url=args.broker + ) + try: + await measure(client, "orbit eventing", args.messages, args.burst, + args.payload) + finally: + await client.close() + + +async def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("which", nargs="?", default="both", + choices=["zmq", "orbit", "both"]) + parser.add_argument("--broker", default=os.environ.get( + "RADICAL_ORBIT_BROKER_URL"), + help="ORBIT broker URL (default: ORBIT's own resolution)") + parser.add_argument("--messages", type=int, default=N_MEAS) + parser.add_argument("--burst", type=int, default=N_BURST) + parser.add_argument("--payload", type=int, default=0, + help="payload size in bytes (0: a bare int)") + + args = parser.parse_args() + + if args.which in ("zmq", "both"): + await bench_zmq(args) + + if args.which in ("orbit", "both"): + await bench_orbit(args) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/src/digitaltwin/__init__.py b/src/digitaltwin/__init__.py index d882b6c..dc541d0 100644 --- a/src/digitaltwin/__init__.py +++ b/src/digitaltwin/__init__.py @@ -13,7 +13,7 @@ WindowDataType, WindowedTypeData, ) -from .config import stream_addresses +from .config import stream_addresses, stream_backend from .runtime import DTRuntime, RuntimeAPI, RuntimeState from .streaming import ( CODEC_CLOUDPICKLE, @@ -56,4 +56,5 @@ "ZMQ_PS_Client", "connect_stream_client", "stream_addresses", + "stream_backend", ] diff --git a/src/digitaltwin/config.py b/src/digitaltwin/config.py index b43292a..65a674c 100644 --- a/src/digitaltwin/config.py +++ b/src/digitaltwin/config.py @@ -1,13 +1,22 @@ """Deployment configuration for the digital twin framework. -The single place where stream endpoint addresses are decided. No other -module (and no demo) may contain a hardcoded transport address. - -Binding policy: the framework binds to loopback unless the deployment -explicitly configures something else. The pubsub data plane carries -cloudpickled payloads, so anyone who can reach the broker ports can -execute code in every subscriber -- non-loopback binds require an -explicit configuration and a firewalled/private network. +The single place where the stream transport is decided: which backend +carries the data plane, and -- for the ZMQ one -- at which addresses. No +other module (and no demo) may contain a hardcoded transport address. + +Backend choice (`DT_STREAM_BACKEND`) is a deployment-time decision at the +same altitude as the compute backend; nothing above `PubSubBackend` +depends on which one is in use. + +- `zmq` (default): the framework's own XSUB/XPUB broker. Local and + two-terminal use. Its payloads are cloudpickled and its ports carry no + authentication, so anyone who can reach them can execute code in every + subscriber (risk R7). Hence the binding policy: loopback unless the + deployment explicitly configures something else, and a non-loopback + bind requires a firewalled/private network. +- `orbit`: ORBIT eventing. The same cloudpickled payloads, but inside + the token-authenticated WS star -- no DT-owned ports at all. This is + what closes R7 for a deployment, and it is required before production. """ import os @@ -27,6 +36,38 @@ ENV_PUB_ADDR = "DT_STREAM_PUB_ADDR" ENV_SUB_ADDR = "DT_STREAM_SUB_ADDR" +# which transport carries the data plane -- see the module docstring +BACKEND_ZMQ = "zmq" +BACKEND_ORBIT = "orbit" +STREAM_BACKENDS = (BACKEND_ZMQ, BACKEND_ORBIT) + +# zmq: the local/two-terminal default. A deployment that wants R7 closed +# selects 'orbit' -- deliberately, not by accident of the environment. +DEFAULT_STREAM_BACKEND = BACKEND_ZMQ + +ENV_STREAM_BACKEND = "DT_STREAM_BACKEND" + + +def stream_backend(name: str | None = None) -> str: + """Resolve which pubsub backend carries the data plane. + + Precedence: explicit argument, then `DT_STREAM_BACKEND`, then `zmq`. + An unknown name is an error rather than a silent fallback -- a typo + must not quietly reopen the ZMQ ports of a deployment that asked for + the token-authenticated one. + """ + + chosen = (name or os.environ.get(ENV_STREAM_BACKEND) + or DEFAULT_STREAM_BACKEND).strip().lower() + + if chosen not in STREAM_BACKENDS: + raise ValueError( + f"unknown stream backend {chosen!r};" + f" expected one of {', '.join(STREAM_BACKENDS)}" + ) + + return chosen + def stream_addresses( pub_addr: str | None = None, sub_addr: str | None = None diff --git a/src/digitaltwin/service/plugin.py b/src/digitaltwin/service/plugin.py index 60146f0..c5c31c7 100644 --- a/src/digitaltwin/service/plugin.py +++ b/src/digitaltwin/service/plugin.py @@ -11,11 +11,17 @@ - the wire: one graph verb per `twin_call`, cloudpickle-base64 payloads with a version stamp, and `twin_list` as the only observation path. -**Binding policy (risk R7)**: the stream broker binds loopback by -default. Its payloads are cloudpickled, so anyone who can reach its -ports executes code in every subscriber. A non-loopback bind needs an -explicit `DT_STREAM_PUB_ADDR` / `DT_STREAM_SUB_ADDR` configuration *and* -a firewalled/private network. +**Data plane (risk R7)**: `DT_STREAM_BACKEND` picks the transport. + +- `orbit` puts the twins' streams inside the same token-authenticated + WebSocket star as the control plane, and the embedded ZMQ broker is + then never started -- no DT-owned ports exist at all. This is what + closes R7, and it is what a production deployment selects. +- `zmq` (the default) runs the embedded stream broker, and *binds + loopback*. Its payloads are cloudpickled, so anyone who can reach its + ports executes code in every subscriber. A non-loopback bind needs an + explicit `DT_STREAM_PUB_ADDR` / `DT_STREAM_SUB_ADDR` configuration + *and* a firewalled/private network. """ import asyncio @@ -30,8 +36,18 @@ from radical.orbit.plugin_base import Plugin from starlette.requests import Request -from ..config import embedded_stream_addresses -from ..streaming import ZMQ_BrokerProcess +from ..config import ( + BACKEND_ORBIT, + BACKEND_ZMQ, + embedded_stream_addresses, + stream_backend, +) +from ..streaming import ( + CLIENT_CONNECT_TIMEOUT, + PubSubClient, + ZMQ_BrokerProcess, + connect_stream_client, +) from .client import DTClient from .session import VERBS, DTSession @@ -82,8 +98,13 @@ def __init__(self, app: FastAPI, instance_name: str = "dt"): self.broker_url: Optional[str] = os.environ.get(ENV_BROKER_URL) or None + # which transport carries the twins' streams. Resolved once, at + # plugin construction: a deployment decision, not a per-twin one. + self.stream_backend: str = stream_backend() + # the embedded DT stream broker, shared plugin-wide and started on - # first need (see `stream_addresses`) + # first need (see `stream_addresses`). Never started at all under + # the 'orbit' backend -- that is the point of it. self._stream_broker: Optional[ZMQ_BrokerProcess] = None self._stream_addrs: Optional[tuple[str, str]] = None self._stream_lock = asyncio.Lock() @@ -254,13 +275,7 @@ async def admin_sessions(self, request: Request) -> dict: entry["sid"] = sid sessions.append(entry) - return { - "sessions": sessions, - "stream_broker": { - "addresses": self._stream_addrs, - "alive": bool(self._stream_broker and self._stream_broker.is_alive()), - }, - } + return {"sessions": sessions, "stream_broker": self.stream_summary()} # -- observability ------------------------------------------------------ @@ -309,6 +324,49 @@ async def on_topology_change(self, participants: dict) -> None: sid, ", ".join(sorted(lost)), ", ".join(failed), ) + # -- the data plane ----------------------------------------------------- + + async def connect_stream( + self, namespace: str, timeout: Optional[float] = CLIENT_CONNECT_TIMEOUT + ) -> PubSubClient: + """A connected, namespaced stream client on the selected backend. + + The single place where the plugin's transport choice is applied. + The `orbit` branch never touches `stream_addresses`, which is what + keeps the embedded ZMQ broker from being started in a deployment + that asked for the token-authenticated data plane. + """ + + if self.stream_backend == BACKEND_ORBIT: + return await connect_stream_client( + namespace, + timeout=timeout, + backend=BACKEND_ORBIT, + broker_url=self.broker_url, + ) + + pub_addr, sub_addr = await self.stream_addresses() + + # named explicitly: the choice was resolved once at construction, + # and re-reading the environment per twin could contradict it + return await connect_stream_client( + namespace, pub_addr, sub_addr, timeout, backend=BACKEND_ZMQ + ) + + def stream_summary(self) -> dict: + """The data plane's entry in `admin/sessions`.""" + + if self.stream_backend == BACKEND_ORBIT: + # no addresses and nothing to supervise: the streams ride the + # ORBIT connection the control plane already has + return {"backend": BACKEND_ORBIT} + + return { + "backend": self.stream_backend, + "addresses": self._stream_addrs, + "alive": bool(self._stream_broker and self._stream_broker.is_alive()), + } + # -- embedded stream broker -------------------------------------------- async def stream_addresses(self) -> tuple[str, str]: diff --git a/src/digitaltwin/service/session.py b/src/digitaltwin/service/session.py index 4a69473..bf29d4f 100644 --- a/src/digitaltwin/service/session.py +++ b/src/digitaltwin/service/session.py @@ -23,7 +23,7 @@ from ..components import DataType, TypedData from ..runtime import DTRuntime, RuntimeState -from ..streaming import PubSubClient, connect_stream_client +from ..streaming import PubSubClient from .wire import Package, check_versions, decode, encode log = logging.getLogger("radical.orbit") @@ -669,9 +669,10 @@ async def _init_twin(self, twin: TwinInstance) -> None: flow, *_ = await asyncio.gather(*map(self.engine, names)) - pub_addr, sub_addr = await self._plugin.stream_addresses() - stream = await connect_stream_client( - twin.twin_id, pub_addr, sub_addr, STREAM_CONNECT_TIMEOUT + # the plugin owns the transport choice (zmq / orbit); the + # twin only ever sees a connected, namespaced client + stream = await self._plugin.connect_stream( + twin.twin_id, STREAM_CONNECT_TIMEOUT ) twin.ready(DTRuntime(flow, stream), stream) diff --git a/src/digitaltwin/streaming.py b/src/digitaltwin/streaming.py index 86da102..b453405 100644 --- a/src/digitaltwin/streaming.py +++ b/src/digitaltwin/streaming.py @@ -40,7 +40,13 @@ from zmq.utils.monitor import recv_monitor_message from .components import DataType, TypedData -from .config import RANDOM_PUB_ADDR, RANDOM_SUB_ADDR, stream_addresses +from .config import ( + BACKEND_ORBIT, + RANDOM_PUB_ADDR, + RANDOM_SUB_ADDR, + stream_addresses, + stream_backend, +) logger = logging.getLogger(__name__) @@ -96,12 +102,17 @@ def decode_payload(payload: bytes, codec: str): class PubSubBackend(ABC): - """Abstract base class for publish/subscribe backends. + """The transport seam: everything above this is transport-agnostic. - Subclasses must implement the asynchronous ``connect`` method and the - core publish/subscribe operations. + A backend delivers to per-topic subscriber callbacks from a single + receive loop it owns. That loop, the subscriber registry and the + closed/running state are the same in every backend, so they live + here; a subclass supplies `connect`, `publish`, `subscribe`, + `unsubscribe`, `close` and a `_run()` body. """ + label = "generic" + # names this backend in a PubSubConfig: what has to reopen the # endpoint. Every backend declares its own. kind = "generic" @@ -117,12 +128,102 @@ def __init__(self): # the failure mode this exists to prevent. self.on_error: Optional[Callable[[BaseException], None]] = None + # topic -> subscriber callbacks. Delivery is filtered by exact + # topic lookup, whatever the transport matched on the wire. + self.topics: dict[str, list[Callable]] = {} + + # topics whose payload the transport hands over untouched: + # something above the seam owns their wire format (see the codecs) + self.raw_topics: set[str] = set() + + self._task: Optional[asyncio.Task] = None + self._closed = False + self.is_running = asyncio.Event() + def _report_error(self, exc: BaseException): logger.error("stream backend failed: %s", exc, exc_info=exc) if self.on_error is not None: self.on_error(exc) + def _check_open(self): + if self._closed: + raise RuntimeError("stream client is closed") + + async def _await_running(self, what: str): + """Callers which arrive before `connect()` finished are made to + wait rather than to lose their message. + + The re-check afterwards matters: a client closed while somebody + was waiting here must produce the ordinary closed-client error, + not an attribute error somewhere in a half-dismantled backend. + """ + + if not self.is_running.is_set(): + logger.warning("requesting %s before connecting to broker. Waiting", + what) + await self.is_running.wait() + self._check_open() + + def _stop_running(self): + """Mark the backend disconnected, waking anyone parked in + `_await_running` on the way. + + A waiter is otherwise stranded: it parked on a connect that has + now been abandoned, and a plain `clear()` would leave it waiting + for one that will never arrive. `set()` resolves the waiters' + futures immediately and the `clear()` right after does not + un-resolve them, so they wake into `_check_open` and get the + ordinary closed-client error. + """ + + self.is_running.set() + self.is_running.clear() + + def _start_receiving(self): + """Arm the supervised receive loop. Called at the end of connect.""" + + self._task = asyncio.create_task(self._run()) + self._task.add_done_callback(self._run_done) + + async def _dispatch(self, topic: str, message): + """Hand one decoded message to the topic's subscribers. + + One failing subscriber must not starve its siblings + (`CancelledError` is not an `Exception`: close() still wins). + """ + + for task in self.topics.get(topic, []): + try: + await task(message) + except Exception: + logger.exception("subscriber failed on topic %r", topic) + + async def _cancel_receiving(self): + """Cancel and await the receive loop. Part of every close().""" + + task, self._task = self._task, None + if task is not None: + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + def _run_done(self, task: asyncio.Task): + """The receive loop only ends on close(). Any other exit means the + twin stopped receiving -- report it instead of stalling silently.""" + + if self._closed or task.cancelled(): + return + + exc = task.exception() or RuntimeError("stream receive loop exited") + self._report_error(exc) + + async def _run(self): + """Receive loop: decode frames and `_dispatch` them. Runs until + cancelled by close().""" + + raise NotImplementedError + @abstractmethod async def connect(self, *args: Any, **kwargs: Any) -> None: """Connect the backend to the message broker. @@ -357,18 +458,6 @@ def __init__( self._ctx.socket(zmq.SUB) if sub_addr is not None else None ) - # subscribe: store the callback for the topic - # publish: send a message to each of the callbacks. - - self.topics: dict[str, list[Callable]] = {} - - # topics whose payload the transport must hand over untouched: - # something above the seam owns their wire format (see the codecs) - self.raw_topics: set[str] = set() - - self._task: Optional[asyncio.Task] = None - self._closed = False - self.is_running = asyncio.Event() async def _connect_socket(self, sock, addr): """Connect `sock` and wait until the connection is established. @@ -423,8 +512,7 @@ async def connect(self, timeout: Optional[float] = CLIENT_CONNECT_TIMEOUT): raise if self.sub_soc is not None: - self._task = asyncio.create_task(self._run()) - self._task.add_done_callback(self._run_done) + self._start_receiving() self.is_running.set() @@ -438,9 +526,7 @@ async def publish(self, topic, message, raw=False): if self.pub_soc is None: raise ValueError("Publishing endpoint not connected") - if not self.is_running.is_set(): - logger.warning("Requesting publish before connecting to broker. Waiting") - await self.is_running.wait() + await self._await_running("publish") topic_b = topic.encode("utf-8") message_b = message if raw else cloudpickle.dumps(message) @@ -456,9 +542,7 @@ async def subscribe(self, topic, callback, raw=False, **backend_params): if self.sub_soc is None: raise ValueError("Subscribe endpoint not connected") - if not self.is_running.is_set(): - logger.warning("Requesting subscribe before connecting to broker. Waiting") - await self.is_running.wait() + await self._await_running("subscribe") self.sub_soc.setsockopt(zmq.SUBSCRIBE, topic.encode("utf-8")) @@ -485,12 +569,8 @@ async def close(self): return self._closed = True - task, self._task = self._task, None try: - if task is not None: - task.cancel() - with contextlib.suppress(asyncio.CancelledError): - await task + await self._cancel_receiving() finally: # the guard above makes close() a one-shot, so the sockets and # the context have to go even if the await was cancelled @@ -504,11 +584,7 @@ async def close(self): # all sockets are closed, so this returns immediately self._ctx.term() - self.is_running.clear() - - def _check_open(self): - if self._closed: - raise RuntimeError("stream client is closed") + self._stop_running() async def _run(self): """Receive loop. A single bad payload or a raising callback must @@ -527,23 +603,7 @@ async def _run(self): logger.exception("dropping malformed message: %r", frames[:1]) continue - for task in self.topics.get(item, []): - # one failing subscriber must not starve its siblings - # (CancelledError is not an Exception: close() still wins) - try: - await task(data) - except Exception: - logger.exception("subscriber failed on topic %r", item) - - def _run_done(self, task: asyncio.Task): - """The receive loop only ends on close(). Any other exit means the - twin stopped receiving -- report it instead of stalling silently.""" - - if self._closed or task.cancelled(): - return - - exc = task.exception() or RuntimeError("stream receive loop exited") - self._report_error(exc) + await self._dispatch(item, data) # The pubsub client abstracts away the specifics of the pub / sub @@ -606,6 +666,7 @@ def config(self) -> "PubSubConfig": pub_addr=self._backend.pub_addr, sub_addr=self._backend.sub_addr, kind=self._backend.kind, + broker_url=getattr(self._backend, "broker_url", None), ) def topic(self, dtype: DataType) -> str: @@ -767,12 +828,17 @@ class PubSubConfig: `kind` names the backend which has to open the endpoint; a backend declares its own (`PubSubBackend.kind`). It stays a plain string so a further backend needs no change here. + + The address fields belong to the `zmq` kind; `broker_url` to the + `orbit` kind (`None` uses ORBIT's own resolution, and the auth token + is deliberately not part of the config -- it is resolved locally). """ namespace: Optional[str] = None pub_addr: Optional[str] = None sub_addr: Optional[str] = None kind: str = ZMQ_PS_Client.kind + broker_url: Optional[str] = None @classmethod def resolve( @@ -797,13 +863,22 @@ async def connect_backend(self, timeout: Optional[float] = None) -> PubSubBacken broker leaks neither sockets nor a context. """ - if self.kind != ZMQ_PS_Client.kind: + if self.kind == BACKEND_ORBIT: + # imported here: `radical.orbit` is the optional 'service' + # extra, and a plain ZMQ install must not need it + from .streaming_orbit import OrbitPubSubBackend + + backend = OrbitPubSubBackend(self.broker_url) + + elif self.kind == ZMQ_PS_Client.kind: + backend = ZMQ_PS_Client(self.pub_addr, self.sub_addr) + + else: raise ValueError( f"cannot open a {self.kind!r} stream endpoint here:" f" no backend of that kind is available" ) - backend = ZMQ_PS_Client(self.pub_addr, self.sub_addr) await backend.connect(timeout) return backend @@ -871,11 +946,26 @@ async def connect_stream_client( pub_addr: Optional[str] = None, sub_addr: Optional[str] = None, timeout: Optional[float] = CLIENT_CONNECT_TIMEOUT, + *, + backend: Optional[str] = None, + broker_url: Optional[str] = None, ) -> PubSubClient: """Build and connect a namespaced stream client from configuration. - The connect is bounded by `timeout` -- see `ZMQ_PS_Client.connect`. + `backend` selects the transport (`zmq` / `orbit`); unset, it comes + from `DT_STREAM_BACKEND` (see `config.stream_backend`). The address + arguments belong to the ZMQ backend and `broker_url` to the ORBIT one + -- each is ignored by the other, and both default to their own + resolution. + + The connect is bounded by `timeout` in either case. """ - cfg = PubSubConfig.resolve(namespace, pub_addr, sub_addr) + name = stream_backend(backend) + + if name == BACKEND_ORBIT: + cfg = PubSubConfig(namespace, kind=BACKEND_ORBIT, broker_url=broker_url) + else: + cfg = PubSubConfig.resolve(namespace, pub_addr, sub_addr) + return await cfg.connect(timeout) diff --git a/src/digitaltwin/streaming_orbit.py b/src/digitaltwin/streaming_orbit.py new file mode 100644 index 0000000..58283f4 --- /dev/null +++ b/src/digitaltwin/streaming_orbit.py @@ -0,0 +1,389 @@ +"""The DT data plane on ORBIT eventing (DTaaS plan M3, risk R7). + +A second `PubSubBackend` behind the M0 seam. Nothing above the seam +changes: `PubSubClient` still hands topics down and gets decoded messages +back, and a persistent component still publishes through +`RuntimeAPI.stream`. What changes is where the bytes travel. + +Why this exists +--------------- +The ZMQ backend's payloads are cloudpickled and its XSUB/XPUB ports carry +no authentication: anyone who can reach them executes code in every +subscriber. This backend moves exactly the same payloads inside ORBIT's +token-authenticated WebSocket star, so the data plane is no weaker than +the control plane and the deployment opens no DT-owned ports at all. +The payloads are *still* cloudpickle -- the win is the trust boundary +around them, which is the same one that already accepts client-shipped +component classes (plan risk R4). + +How it maps +----------- +- **Publish** is `EndpointRuntime.send_notification(plugin, topic, + data)`: a plain sync, non-blocking call that emits one `event` frame. + The DT topic (`dt//dtypes/