feat: Add breakwater-deich: A distributed server#97
Open
sbernauer wants to merge 17 commits into
Open
Conversation
Rewrite the distributed collector/worker around the fact that the
persistent canvas already merges by absolute per-pixel timestamp
(last-write-wins), which is commutative — so cross-node frame ordering and
slot alignment are unnecessary. Each worker sends a full framebuffer that
supersedes its earlier ones, so the collector need only keep each worker's
latest frame and fold them all in on a render timer.
This deletes the entire windowing subsystem: the shared wall-clock
FrameSchedule, per-frame numbering, the FrameStore + interest window, the
render margin, and late/future-frame classification — along with their
cross-node clock coupling and self-inflicted edge cases (e.g. the --fps 1
"frame still in flight" case the margin existed to patch).
Structure:
- protocol.rs (was sync.rs): message types + a Connection<S> that owns the
wire format and the "marker then raw blob" framebuffer contract, so no
caller re-implements it. Adds a codec round-trip test over a duplex.
- collector/{mod,canvas,stats}.rs (was collector.rs): mod orchestrates,
canvas is the timestamped LWW merge, stats is the per-IP aggregator with
its cross-crate monotonicity contract now documented explicitly.
- worker.rs: pushes on a local timer (no frame numbers); MissedTickBehavior
::Delay avoids burst/backpressure spirals on slow pushes.
- collector: all support tasks keep their JoinHandles (no detached tasks);
ordered shutdown with the stats drain aborted last.
Behavior preserved: canvas persistence across gaps/restarts, blank frame
never clobbers live content, duplicate-worker refusal, statistics delta
folding + save/restore.
Verified locally with a collector + two workers: both connect, their frames
merge (10 + 25 px drawn → 35 live px), and a killed worker's pixels persist.
9 tests pass; clippy (pedantic = deny) clean.
Worker/collector observability to validate the fleet, kept entirely off the per-pixel hot path — everything is measured once per frame (≈ fps × workers/s), a couple of clock reads plus a histogram observe: - worker: framebuffer send-duration histogram + a lagged-frames counter (whole frame-periods a single send overran, i.e. FPS we couldn't sustain) - collector: per-worker receive-duration and end-to-end latency histograms, plus a total ingress-bytes counter for link-utilisation headroom - end-to-end latency needs a send-start timestamp, so WorkerMessage::Framebuffer now carries an 8-byte sent_at_ns (bumps the wire format; deploy nodes together) Exposed both ways: Prometheus is enabled by default (like breakwater) — each node serves /metrics on --prometheus-listen-address (default [::]:9100, a shared CLI arg flattened into both roles) — and every node logs a per-interval summary too. Metrics live in a self-contained `metrics` module (shared exporter, timing buckets and CLI arg; the two roles keep their own metric sets); hooks are small additions to worker/collector/protocol. Verified with a collector + two workers: /metrics and the log summaries populate with per-worker labels, latency, and ingress.
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.