SWITCHYARD — Downstream Correction-Budget Planner

“Publishing ten seconds earlier created eleven downstream corrections and left a stale snapshot in the archive. Here is the exact revision bill.”

SWITCHYARD is a deterministic streaming replay engine and downstream correction-budget planner. Rather than treating stream processing as an abstract animated watermark or raw throughput benchmark, SWITCHYARD explores the operational tension between Publication Latency and Downstream Revision Burden.

It models dual-clock temporal windowing, partition candidate watermarks with simulated idle detection, and computes the exact operational bill incurred across consumers with diverse SLA contracts (revisable vs. immutable, fan-out, and latency).

---

1. Key Workspaces & Capabilities

Workspace	Purpose	Signature Deliverable
Stream & Time Plot	Dual-clock 2D event replay	SVG scatter plot with 45° causality line ($t_A \ge t_E$), tumbling window bands, monotonic watermark tracking, and scrubbing controls.
Correction Ledger	Downstream revision accounting	Computes the operational work units and stale snapshots created across consumers when late arrivals revise window totals.
Window Inspector	In-flight window state machine	Drill-down into window intervals $[W_{\text{start}}, W_{\text{end}})$, status progression (OPEN, EMITTED, REVISABLE, FINALIZED), and late rejects.
Checkpoint Lab	State snapshot & recovery verification	Captures complete engine state; proves 100% restored equivalence vs. uninterrupted replay; features a teaching mode demonstrating deduplication state loss.
Trade-off Benchmark	Empirical Pareto frontier	60-trial benchmark across 20 seeds comparing aggressive early publication, moderate windowing, and generous lateness regimes.
Export & Audit	Reproducible bundles & incident briefs	Downloadable JSON run bundles with SHA-256 event checksums, executable in CI, and printable incident reports.

---

2. Interactive Screenshots Gallery

1. Stream & Time Plot: Dual-Clock Replay

2D visual mapping of event time ($t_E$) against arrival time ($t_A$) with horizontal tumbling window bands and dynamic watermark progression:

2. Downstream Correction Ledger (Signature Centerpiece)

Computes the exact revision bill (11 work units and 1 stale snapshot) across revisable wallboards, immutable archives, and fan-out summaries:

3. Tumbling Window Inspector

Detailed inspection of window intervals, accumulated in-flight totals, initial emissions, revisions, and the too-late rejection ledger:

4. Checkpoint Lab: State Recovery Invariant

Validates state snapshotting, restoration equivalence, and features an interactive teaching mode demonstrating duplicate leaks when dedup state is dropped:

5. Empirical Policy Benchmark & Interactive Sandbox

Evaluates the trade-off frontier across 60 trials and provides interactive sliders to test custom windowing parameters in real time:

Responsive Viewports: Tablet & Mobile

Tested across desktop (1440px), tablet (1024px), and mobile (390px) viewports with accessible touch navigation:

Tablet Layout (1024px)	Mobile Layout (390px)

---

3. Mathematical Foundations & Canonical Proof

The Canonical 2-to-5 Revision Fixture

In our verified test fixture:

• Tumbling Window Width: $300\text{ s}$ ($[0, 300\text{ s})$)
• Out-of-Order Bound: $0\text{ s}$
• Allowed Lateness: $120\text{ s}$ (finalization bound at $420\text{ s}$)

Execution Timeline:

1. $e_1$ ($t_E=10\text{ s}, t_A=10\text{ s}, \text{val}=2$): In-flight sum $= 2$. Window status: OPEN.
2. $e_2$ ($t_E=310\text{ s}, t_A=310\text{ s}, \text{val}=1$): Global watermark reaches $310\text{ s} \ge 300\text{ s}$.
   • Window $[0, 300\text{ s})$ emits initial value $V_0 = 2$.
   • wallboard (delay 0s): publishes 2 at $t_A = 310\text{ s}$.
   • archive (delay 0s): publishes 2 at $t_A = 310\text{ s}$.
   • summary (delay 10s): schedules publication for $t_A = 320\text{ s}$.
3. $e_3$ ($t_E=20\text{ s}, t_A=320\text{ s}, \text{val}=3$):
   • Arrives while $WM = 310\text{ s} < 420\text{ s}$ (within allowed lateness).
   • Window sum updates to $2 + 3 = 5$.
   • REVISE_UNTIL_FINALIZED triggers REVISION emission of value $5$.
   • Downstream Bill:
     • wallboard ($F=1$, revisable): issues correction $\to 1\text{ work unit}$.
     • archive ($F=1$, immutable): rejects update $\to 1\text{ stale snapshot}, 0\text{ work units}$.
     • summary ($F=10$, revisable): published at $t_A=320\text{ s}$, issues fan-out correction $\to 10\text{ work units}$.
     • Reconciled Work Units: $1 + 0 + 10 = \mathbf{11\text{ units}}$.
     • Reconciled Stale Snapshots: $\mathbf{1}$.

The Patient Alternative:

If a publication delay of $120\text{ s}$ is configured (delaying publication until window finalization), the window emits only once at $t_A = 420\text{ s}$ with the final value of $5$, incurring 0 correction work units and 0 stale snapshots at the expense of $120\text{ s}$ publication delay.

---

4. Empirical Benchmark

Measured across 60 experimental trials (20 seeds × 3 policy regimes, 500 events per stream) via scripts/run_benchmark.mjs:

Policy Configuration	Mean Correction Work Units	Mean Stale Snapshots	Mean Late Rejections	Mean Window Error vs Oracle	Mean Run Latency
Aggressive Early Publication (Zero Lateness)	0.0 units	0.0	13.8 rows	6.95 count	1.54 ms
Moderate Windowing (15s Bound, 60s Lateness)	45.5 units	5.5	5.2 rows	2.93 count	1.11 ms
Generous Lateness (30s Bound, 180s Lateness)	82.0 units	8.4	0.1 rows	0.07 count	1.19 ms

• Total Benchmark Runtime: 0.100 seconds for all 60 trials.
• Engine Throughput: ~300,000 events/second in client-side in-memory JavaScript.

Full methodology and per-trial output: docs/BENCHMARK.md.

---

5. Prior Art & Attribution

SWITCHYARD builds upon foundational stream processing literature and does not claim to have invented event-time windowing or watermark mechanics:

Pioneer System	Foundation Relevant to SWITCHYARD	SWITCHYARD Focus
Apache Beam / Dataflow Model	Unified event-time windowing, watermarks, triggers, and allowed lateness.	Translates abstract triggers into an inspectable downstream correction ledger.
Apache Flink	Stateful stream processing, asynchronous checkpointing, and unaligned state recovery.	Provides a client-side checkpoint recovery lab with interactive deduplication loss demonstration.
Spark Structured Streaming	Watermark-based state eviction and append vs. update modes.	Visualizes the Pareto frontier between publication latency and revision overhead.

Full attribution: docs/PRIOR_ART.md.

---

6. Quickstart & Local Setup

Prerequisites

• Node.js $\ge 18$ (tested on Node v20/v24)

Installation

# Navigate to project directory
cd d:\Ran\switchyard

# Install dependencies
npm install

Run Automated Contract Tests

npm test

Executes all 5 stream contract tests in ~130ms.

Start Local Development Server

npm run dev

Launches Vite dev server at http://localhost:5174/.

Build Production Bundle

npm run build

Compiles optimized static assets in dist/ (JavaScript: 64.7 kB gzipped, CSS: 540 bytes).

Run Benchmark Suite

npm run benchmark

Executes 60 trials across 20 seeds in ~100ms and updates docs/BENCHMARK.md.

---

7. Project Documentation Index

• docs/ENGINE_CONTRACT.md: Dual-clock grammar, watermark rules, window lifecycle states, consumer SLA definitions.
• docs/PRIOR_ART.md: Attribution of related streaming literature and open-source systems.
• docs/BENCHMARK.md: 60-trial empirical benchmark report across 20 seeds.
• docs/VALIDATION.md: Automated verification run log and mathematical proofs.
• docs/PERFORMANCE.md: Runtime latency, memory bounds, and bundle budget profile.
• docs/DEMO_SCRIPT.md: 60-second executive walkthrough script.
• docs/INTERVIEW_WALKTHROUGH.md: Senior/Staff system design interview preparation guide.
• docs/LINKEDIN_POST.md: Ready-to-publish engineering launch draft.
• BUILD_STATUS.md: Complete milestone progress tracker (100% complete).