From 40ba6a5eb229ec93ed5237d670a1eed6cef6b5f1 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 05:53:59 +0000 Subject: [PATCH 1/5] docs: replace dev-process artifacts with user-facing API reference and roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Delete the build-process ledger and internal planning docs (PROGRESS.md, SRS, implementation plan, agent prompt) — development history, not useful to library users. In their place: - docs/API.md: consolidated C ABI reference (all functions, error codes, lifecycle/sizing rules, zero-copy borrow rules, segment layout and flags contract, liveness caveat, Python/Rust binding patterns) — closes the long-standing NFR-M2 documentation gap. - docs/ROADMAP.md: triaged upgrade directions (already-shipped vs v1.x candidates vs exploratory). Makefile/ci.yml comments that pointed at PROGRESS.md now carry the explanation inline. .gitignore gains defensive entries for common editor/toolchain droppings. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01JykzcEronyifSN4RnTgjTk --- .github/workflows/ci.yml | 6 +- .gitignore | 6 + Makefile | 4 +- PROGRESS.md | 164 ------------- docs/API.md | 360 ++++++++++++++++++++++++++++ docs/ROADMAP.md | 61 +++++ docs/SHUTTLE_AGENT_PROMPT.md | 203 ---------------- docs/Shuttle_Implementation_Plan.md | 210 ---------------- docs/Shuttle_SRS.md | 331 ------------------------- 9 files changed, 432 insertions(+), 913 deletions(-) delete mode 100644 PROGRESS.md create mode 100644 docs/API.md create mode 100644 docs/ROADMAP.md delete mode 100644 docs/SHUTTLE_AGENT_PROMPT.md delete mode 100644 docs/Shuttle_Implementation_Plan.md delete mode 100644 docs/Shuttle_SRS.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a2a3c2b..313bf8b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,9 +4,9 @@ # Performance-MEASUREMENT tests are excluded here (-E filter below): they # assert latency percentiles / CPU ratios that are only meaningful on # controlled hardware, and shared CI runners produce arbitrary numbers. -# Those gates were verified on the project's two-platform harness (see -# PROGRESS.md); CI guards correctness, lifecycle, crash recovery, the -# lock-free protocol, and the cross-language FFI. +# Those gates were verified on the project's two-platform harness; CI guards +# correctness, lifecycle, crash recovery, the lock-free protocol, and the +# cross-language FFI. name: CI on: diff --git a/.gitignore b/.gitignore index 1921b24..a4c810d 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,9 @@ .DS_Store build/ build-*/ +__pycache__/ +*.pyc +target/ +.vscode/ +.idea/ +CMakeUserPresets.json diff --git a/Makefile b/Makefile index 0439482..8b6e618 100644 --- a/Makefile +++ b/Makefile @@ -8,8 +8,8 @@ # segments implied by 50 MB payloads + the SRS 2x capacity rule; ftruncate/ # mmap would fail with a confusing ENOSPC/EINVAL. # -# setarch -R on linux TSan runs: gcc-13 TSan vs Ubuntu 24.04 ASLR entropy -# workaround (documented in PROGRESS.md; pre-approved, not a suppression). +# setarch -R on linux TSan runs: gcc-13 TSan is incompatible with Ubuntu +# 24.04's default ASLR entropy, so the TSan leg runs under setarch -R. # setarch needs personality(2), which Docker's default seccomp profile blocks # (ADDR_NO_RANDOMIZE arg is not on the allowlist), so the TSan leg runs with # seccomp=unconfined. Dev harness running our own code only. diff --git a/PROGRESS.md b/PROGRESS.md deleted file mode 100644 index e6079f4..0000000 --- a/PROGRESS.md +++ /dev/null @@ -1,164 +0,0 @@ -# Shuttle Build Ledger - -## BUILD COMPLETE — all 27 gates PASS on every applicable leg (2026-06-11) - -Phases 0–7 done in 26 loop iterations over three days (2026-06-09 → 06-11), -zero escalations, no BLOCKED.md ever written. Final suite: 28 tests green -under ASan and TSan on macOS (native) and Linux (glibc arm64 container). - -**Headline results** (NFR-P1/P2/P3; labeling per the binding amendments): -- 50 MB end-to-end latency: 5 µs median native macOS (vs UDS 9.3 ms = 1857×, - HTTP 8.5 ms = 1699×); 24 µs in the container (482×/541×, VIRTUALIZED — - not headline figures). Requirement was ≥10×, stretch 50×. - **The headline NFR-P1 claim remains PROVISIONAL until run on bare-metal - Linux** — the harness is ready: `make test-linux` on any glibc box. -- Borrow-path copy CPU: 0.03–0.08% of the UDS copy baseline for identical - bytes; borrowed pointers proven in-place within the consumer's mapping. -- Wake latency: p50 3 µs / p99 4 µs under 20k-frames/s load (native); - 59.5/221.8 µs virtualized. Parked-wake (G4.3): p50 7-8 µs. - -**What was built:** C++17 SPSC shared-memory channel — BipBuffer with the -A1 read/write/watermark cursor model, lock-free release/acquire data path -(happens-before contract written inline in spsc.hpp), seq_cst Dekker -parking with bounded timedwaits, heartbeat liveness with kErrPeerDead -(primary on both platforms), robust-mutex recovery on Linux, -os_sync_wait_on_address parking on macOS (ledger decision superseding the -trylock-escape design), frozen C ABI v1 (shuttle_c.h) with Python cffi and -Rust bindings (zero-copy on both; Rust enforces the borrow at compile -time), and the three-transport benchmark harness. - -**Remaining known items (not gate failures):** -- Bare-metal Linux benchmark run to convert the provisional NFR-P1 - headline into a final claim. -- One unexplained single-occurrence linux trickle/latency suite failure on - 2026-06-11 (output lost; 11 clean repeats since; capture instructions in - the decisions log stand if it ever recurs). -- NFR-M2 (single consolidated API/layout/ordering reference doc) is an - S-priority SRS item not covered by any gate; the inline contracts in - spsc.hpp/header.hpp/shuttle_c.h are the current source of truth. - -## Scheduled job: CANCELLED 2026-06-11 (job 2fdb3d70) — all gates pass; the -hourly loop is no longer registered. - -## Current objective: none — build complete - -(Job history: `2fdb3d70`, hourly at :23, created 2026-06-10, cancelled 2026-06-11 on completion.) - -## Gate status - -| Gate | mac leg | linux leg | Verified by | Date | Notes | -|------|---------|-----------|-------------|------|-------| -| G0.1 | PASS | PASS | `make test-mac` / `make test-linux` (ctest 1/1 Passed, ASan+UBSan) | 2026-06-10 | mac leg required CLT 26.5 update — see decisions log | -| G0.2 | PASS | PASS | `make tsan-mac` / `make tsan-linux` (ctest 1/1 Passed under TSan) | 2026-06-10 | linux leg needed seccomp=unconfined for setarch — see decisions log | -| G0.3 | PASS | PASS | `make test-mac` / `make test-linux` (shuttle_shm_smoke: 4 KB + 128 MB page-touched) | 2026-06-10 | Negative control verified: same binary SIGBUSes in container at default 64 MB /dev/shm | -| G0.4 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_pshared_smoke) | 2026-06-10 | macOS pshared condvar park/wake WORKS on 26.5 — no os_sync_wait_on_address fallback needed. posix_spawn role-arg pattern per amendment; also clean under TSan both legs | -| G1.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_lifecycle_test) | 2026-06-10 | Driver creates; spawned child opens + verifies magic/version; bumped version → distinct kErrBadVersion. macOS rounds shm st_size to page size — see decisions log | -| G1.2 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_capacity_test) | 2026-06-10 | Three too-small shapes → kErrCapacityTooSmall; failed create leaves no object; boundary cap == maxp+8 accepted | -| G1.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_leak_test) | 2026-06-10 | Linux: object visible in /dev/shm while alive, gone after unlink. Both: survives close (FR-5), unlinked name → kErrNotFound, double unlink distinct | -| G2.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_bipbuffer_test) | 2026-06-10 | 2×100k pairs (roomy 64KB: 1496 wraps; tight 4KB: 19267 wraps), byte-exact FIFO, size/cursor invariants after every op | -| G2.2 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_bipbuffer_edge_test) | 2026-06-10 | Deterministic scripts w/ hand-computed cursors + pointer-identity wrap proofs; also pins both strict-inequality refusals (wrapped-full vs linear-empty aliasing) | -| G2.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_bipbuffer_oversize_test) | 2026-06-10 | Oversize refused immediately with zero cursor mutation, from empty/linear/wrapped; exact max_payload boundary still fits | -| G3.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_spsc_stress_test + shuttle_spsc_threads_test) | 2026-06-10 | Two-process 40k msgs ≈1.25 GiB byte-exact FIFO (8 s mac / 5 s linux); dual-thread same-code-path config TSan-clean both legs per A2; happens-before argument inline in spsc.hpp | -| G3.2 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_spsc_asym_test) | 2026-06-10 | Both directions byte-exact; fast side asserts ≥500 would-block msgs so the full-spin and empty-spin paths are PROVEN engaged, not incidental | -| G3.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_spsc_wrap_test + _threads variant) | 2026-06-10 | 16 KiB channel, 2–6 KiB payloads: 57,822 wraps / 200k msgs two-process, byte-exact; dual-thread A2 config TSan-clean; wrap count asserted ≥ N/8 (falsifiable) | -| G4.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_park_idle_test) | 2026-06-10 | Idle blocked consumer: 1.4 ms CPU over 2.98 s park (0.05%); 250 ms budget vs ~3 s a spin would burn. All Phase 3 stress gates re-run green over the new parking paths | -| G4.2 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_trickle_test) | 2026-06-11 | 100k random-interval msgs; 0 reads hit the 50 ms threshold (max 7.4 ms) — no wake ever lost to the timedwait backstop; exactly N msgs, byte-exact, none extra | -| G4.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_park_latency_test + hotpath variant) | 2026-06-11 | Parked-wake p99: 39 µs mac (p50 7 µs); 198 µs linux container (p50 85 µs, virtualized — label per amendment). Hot path: 0 park-mutex acquisitions both sides over 10k msgs (instance lock counters) | -| G5.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_crash_heartbeat_test) | 2026-06-11 | SIGKILL mid-reservation+mid-keepalive → victim kErrPeerDead 2.08 s after park (1 s threshold); no premature verdict on live keepaliving peer; uncommitted reservation never surfaces | -| G5.2 | N/A | PASS | `make test-linux`+`tsan-linux` (shuttle_robust_mutex_test; self-skips on mac) | 2026-06-11 | Holder SIGKILLed mid-lock: seam absorbs EOWNERDEAD (repair=documented no-op → consistent), mutex fully usable after. Buggy-recovery scenario proves ENOTRECOVERABLE is detectable (test can fail) | -| G5.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_crash_leak_test) | 2026-06-11 | Producer SIGKILLed → survivor kErrPeerDead → close+unlink → object gone, verified from survivor AND independently from driver (/dev/shm on linux; not-openable both) | -| G5.4 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_crash_mutex_test) | 2026-06-11 | Peer SIGKILLed holding park mutex, survivor parked in blocking API: kErrPeerDead at 2.52 s mac / 2.63 s linux; linux mutex proven recovered+serviceable after. Mac mechanism = os_sync_wait_on_address (decision below), superseding the trylock-escape design | -| G6.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_cabi_python_test) | 2026-06-11 | 1500 msgs byte-exact, C++ producer (via frozen C ABI) → Python cffi consumer, zero-copy memoryview over borrowed ptr; stale borrow raises after release per amendment | -| G6.2 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_cabi_rust_test) | 2026-06-11 | 1500 msgs byte-exact, zero-copy slice verified in place; compile_fail.rs rejected with exactly E0597 (and the valid wrapper compiles, so the failure is meaningful) | -| G6.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_cabi_errors_test) | 2026-06-11 | open-nonexistent → −4, bad capacity → −8, bad name → −1, unlink-missing → −4, empty nonblock read → −12, oversize write → −11; C++/Python/Rust all return ints, nothing thrown/panicked | -| G7.1 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_bench_g71) | 2026-06-11 | 50 MB median: mac-native (dev figures) 5 µs vs UDS 9287/HTTP 8495 µs = 1857×/1699×; linux container (VIRTUALIZED — not headline) 24 µs vs 11628/13056 µs = 482×/541×. Both ≥10× (and ≥50× stretch). Headline NFR-P1 claim remains PROVISIONAL until bare-metal Linux | -| G7.2 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_nocopy_cpu_test) | 2026-06-11 | Consumer CPU for 2 GB via borrow path: 0.22 ms mac / 0.42 ms linux = 0.03% / 0.08% of UDS copy baseline (700 / 506 ms); every borrowed ptr verified inside the consumer's data-region mapping (in-place proof) | -| G7.3 | PASS | PASS | `make test-mac`+`tsan-mac` / `make test-linux`+`tsan-linux` (shuttle_wake_under_load_test) | 2026-06-11 | 20k×16 KB @ ~20k frames/s: p50/p99/max = 3/4/30 µs native, 59.5/221.8/9192 µs virtualized — within the G4.3 budgets (p50<100 µs, p99<1 ms) on both legs | - -## Decisions log - -- 2026-06-09 — Repo `git init` (branch `main`) at `/Users/allannguyen/shuttle`; `.gitignore` covers `.DS_Store` and `build*/`. Removed an empty stray `shuttle/docs/` directory that predated the repo. -- 2026-06-09 — Design amendments A1–A4 + minor amendments in `docs/SHUTTLE_AGENT_PROMPT.md` are binding and override the SRS/plan where they conflict (cursor model = read/write/watermark; TSan gates restructured per A2; heartbeat+timedwait primary on both platforms; seq_cst parking protocol; CLOCK_MONOTONIC condvars; 128-byte padding for hot atomics; G0.4 and G5.4 added; UDS baseline added to Phase 7). - -- 2026-06-10 — **Host-wide install: Command Line Tools for Xcode 26.5** (via `softwareupdate -i`). Reason: the prior CLT (Apple clang 17, clang-1700.4.4.1) shipped an ASan runtime incompatible with macOS 26.5 — ANY ASan-instrumented binary (even a one-line C `main`) spun forever pre-`main` inside `wrap_malloc_default_zone` ← `__malloc_init` (confirmed by `sample`; `MallocNanoZone=0` did not help). CLT 26.5 (Apple clang 21, clang-2100.1.1.101) fixes it. If macOS updates again and ASan binaries start hanging at startup with 100% CPU, suspect this same runtime/OS mismatch first and check `softwareupdate --list` for a newer CLT. -- 2026-06-10 — "Sanitizer presets" realized as Makefile targets + a `SHUTTLE_SAN=off|asan|tsan` CMake cache var with separate build trees (`build/{mac,linux}-{asan,tsan}`), not CMakePresets.json — one command per leg either way, fewer moving parts. ASan build includes UBSan per plan; TSan strictly separate (cannot link both). -- 2026-06-10 — Linux leg = `ubuntu:24.04` glibc 2.39 arm64 image (`docker/Dockerfile`, built as `shuttle-linux-dev`), run with `--shm-size=512m`, repo bind-mounted at `/work`. `tsan-linux` target pre-wires the `setarch -R` ASLR workaround. - -- 2026-06-10 — **Amendment to docs/SHUTTLE_AGENT_PROMPT.md (user-directed, binding):** the G0.4 smoke test and every other multi-process test must launch processes via fork+exec or `posix_spawn` — the driver runs the test binary twice with role arguments. Plain fork without exec is forbidden: TSan on macOS does not support fork-without-exec and the child inherits a broken runtime. - -- 2026-06-10 — **TSan linux leg runs with `--security-opt seccomp=unconfined`.** Docker's default seccomp profile blocks `personality(2)` with ADDR_NO_RANDOMIZE, so the pre-approved `setarch -R` workaround failed with "Operation not permitted". Note: on Docker Desktop's LinuxKit kernel (`vm.mmap_rnd_bits=18`) gcc-13 TSan happens to work even without `setarch`, but we keep the documented workaround functional (via unconfined seccomp on the TSan target only) so the harness also works on stock Ubuntu kernels (`mmap_rnd_bits=32`) where TSan crashes at startup without it. Dev container running our own code; acceptable. - -- 2026-06-10 — **macOS rounds shm object `st_size` up to page size (16 KB on Apple Silicon).** An opener's `fstat` can legitimately see a larger size than the creator's exact `ftruncate` length. Open-time geometry validation (NFR-S2) therefore checks the mapping *covers* the claimed `data_offset + data_capacity` (`>=`), never equality. Any future code deriving capacity from `st_size` instead of the header would be wrong on macOS. - -- 2026-06-11 — **macOS parking switched from pshared condvar to `os_sync_wait_on_address` (14.4+, SHARED flag), per the pre-authorized fallback in the standing orders.** Reason: the trylock-loop-escape design cannot work — a survivor inside `pthread_cond_timedwait` can only return by re-acquiring the mutex, and that internal re-acquisition is a bare lock no trylock loop can protect; a peer SIGKILLed inside its tiny park/wake critical section would strand the survivor forever. Wait-on-address holds nothing (no ownership to die with) and compares the watched value atomically with the sleep (no lost wakeup). Implementation: seam functions `park_wait_cursor`/`park_wake_cursor` — waiters sleep until the peer's cursor (write for consumer, read for producer) changes from a pre-park snapshot; Linux keeps robust mutex + condvar with cursor-recheck-under-lock. Side effect: mac G4.3 wake latency re-measured over os_sync — p50 8 µs, p99 421 µs (was 39 µs via condvar; still well under the 1 ms budget; gate remains PASS on re-run). -- 2026-06-11 — **Open flake observation (single occurrence):** the first linux ASan suite run after the os_sync rework reported shuttle_trickle_test and shuttle_park_latency_test FAILED, but the failure output was lost (LastTest.log overwritten by the verifying re-run before capture). Subsequent evidence: 11 consecutive passes (full suite re-run + direct runs + 4 dedicated repeat rounds), and the new Linux wait guard is mechanically equal-or-stronger than the old (cursor==snapshot recheck under the same lock the waker signals under). NOT suppressed — both tests remain at full strength. If either fails again on any leg: STOP, copy `build//Testing/Temporary/LastTest.log` BEFORE any re-run, and treat as a live lost-wakeup investigation. - -- 2026-06-11 — **Host install: `cffi` 2.0.0 into the user site-packages of the python.org Python 3.13** (`python3 -m pip install --user cffi`) — required by the plan's cffi-over-ctypes preference for the Python binding. Container image gains `python3` + `python3-cffi` (Dockerfile). -- 2026-06-11 — **`libshuttle_c` (the frozen C ABI shared library) is built WITHOUT sanitizers in every preset.** Foreign runtimes (python3) cannot load a sanitizer-instrumented dylib (missing runtime; macOS SIP strips DYLD_INSERT_LIBRARIES from system interpreters, so preloading is not viable). Implemented by moving sanitizer flags onto an INTERFACE target (`shuttle_san`) applied via `link_libraries()` to everything declared after it; `shuttle_c` is declared before and compiles its own copies of the sources. The logic inside is header-only/shared and remains sanitizer-verified through all other targets. NOT a sanitizer suppression — recorded as a build-topology decision. - -- 2026-06-11 — **Rust binding declares the C ABI by hand instead of bindgen** (ten frozen, static_asserted signatures; bindgen would drag libclang into the container for no coverage gain — any drift fails the byte-exact integration test). Container image gains `rustc` via apt (1.75; no cargo — zero-dependency single-file builds via plain rustc). Host uses existing rustup rustc 1.94. Revisit bindgen if/when the ABI grows. - -## Environment verification (iteration zero, 2026-06-09; Docker re-verified 2026-06-10) - -| Check | Result | -|---|---| -| Xcode CLT | OK — `/Library/Developer/CommandLineTools`, Apple clang 17.0.0, target arm64-apple-darwin25.5.0 | -| cmake | OK — 4.3.1 | -| Docker | OK (2026-06-10) — Docker Desktop 29.5.3, server arch arm64 | -| glibc arm64 base image pull | OK (2026-06-10) — `ubuntu:24.04` pulls and runs natively: `uname -m` = aarch64, glibc 2.39 | - -## Session notes (newest first) - -- **2026-06-11 (iteration 26 — G7.3 PASS both legs; ALL GATES PASS — BUILD COMPLETE):** `tests/wake_under_load_test.cpp` (unsanitized -O2): 20,000 measured 16 KB frames at ~20k frames/s offered load (~320 MB/s); consumer drains each frame in ~1 µs and parks between frames, so nearly every receipt is a genuine under-load wake. Native macOS: p50 3 µs / p99 4 µs / max 30 µs — tighter than G4.3's gently-paced 7-8 µs p50 (warm caches under sustained streaming). Container: 59.5/221.8 µs (virtualized label). Both legs within the G4.3 consistency budgets. 28/28 ASan+TSan both legs. Hourly job 2fdb3d70 cancelled per hygiene rules; final summary written at the top of this ledger. Loop terminated: all gates pass. - -- **2026-06-11 (iteration 25 — G7.2 PASS both legs):** NFR-P2 proven via `tests/nocopy_cpu_test.cpp` (unsanitized -O2, same opt-out as the bench): the Shuttle consumer drains 40×50 MB touching two bytes per payload while getrusage accounts its CPU — 0.22 ms total on mac (5.6 µs/msg) and 0.42 ms in the container (10.6 µs/msg) vs the UDS copy baseline's 700/506 ms for identical bytes: **0.03–0.08% of the copy baseline's CPU**, asserted ≤5% (falsifiable: any hidden memcpy in the borrow path trips it). In-place proof: every borrowed pointer asserted within the consumer's own mapping of the data region. **Incident, fully explained:** the first mac TSan suite run had 2 test timeouts (bipbuffer_test, spsc_threads_test). Evidence preserved (/tmp/g72-mac-tsan-LastTest.log): the "hung" single-threaded bipbuffer_test had actually PRINTED ITS SUCCESS OUTPUT before being killed, and host load was 100–156 — an Xcode + iOS 26.5 simulator-runtime install/first-boot was running concurrently on this Mac (CoreSimulator/diskimagesiod processes timestamped exactly to the window). No orphaned test processes found. After load settled (<8), clean re-run: 27/27 with normal timings (10.2 s / 6.8 s). Exogenous host contention, not a product or test bug; no gate weakened. Next objective: **G7.3** — µs-scale wake latency under load, consistent with G4.3 (the final gate). - -- **2026-06-11 (iteration 24 — G7.1 PASS both legs):** Headline benchmark landed in `bench/bench_main.cpp` (rewrote the Phase 0 stub): three transports under one driver — Shuttle borrow path, raw-binary UDS, and a hand-rolled minimal HTTP/1.1 baseline (keep-alive, Content-Length framing, TCP_NODELAY, 4 MB socket buffers — the least wasteful HTTP per D7). Producers fill the whole payload then stamp CLOCK_MONOTONIC into the head, so generation cost is identically excluded; consumers record commit→payload-held deltas; 3 warmup discarded, 20 iters, median+p99. Built UNSANITIZED at -O2 (same opt-out as libshuttle_c; sanitizer overhead would corrupt the ratios) and auto-labels container runs via /.dockerenv. Results: mac-native 50 MB median 5 µs (p99 8) vs UDS 9.29 ms / HTTP 8.50 ms → 1857×/1699×; linux-container (virtualized) 24.1 µs (p99 29.4) vs 11.6 ms / 13.1 ms → 482×/541×. 16 KB stream throughput: shuttle ≈13.6 GB/s vs UDS ≈7 GB/s vs HTTP 0.6–0.9 GB/s. NFR-P1 satisfied against BOTH baselines on both legs; headline claim provisional until bare-metal Linux (labeling per amendment). 26/26 ASan+TSan both legs. Next objective: **G7.2** — profiler evidence that the borrow path spends negligible CPU copying/serializing (NFR-P2): consumer reads producer bytes in place. - -- **2026-06-11 (iteration 23 — G6.3 PASS both legs; PHASE 6 COMPLETE):** Added `tests/cabi_errors_test.cpp` + `tests/ffi/err_probe.py` + `tests/ffi/rust/err_probe.rs`. C++ leg covers six induced errors across the code space (NOT_FOUND, CAPACITY_TOO_SMALL, INVALID_ARGS, WOULD_BLOCK, MSG_TOO_LARGE) through the C ABI; Python (cffi) and Rust (safe wrapper, compiled at test time) each verify the nonexistent-open arrives as integer −4 with clean exit — a raise/panic would exit nonzero, so clean completion IS the no-exception proof. 25/25 ASan+TSan both legs. **Phase 6 done: the ABI is frozen and proven from C++, Python, and Rust.** Next objective: **G7.1** — Phase 7 headline benchmark: 50 MB payload end-to-end latency ≥10× lower than BOTH baselines (raw-binary localhost HTTP per D7, and Unix domain socket per the minor amendment); also the 16 KB frame stream workload; warm-up + many iterations, median and p99; CLOCK_MONOTONIC timestamps in payload. Reminder: numbers measured in Docker are labeled "virtualized — not headline figures"; native macOS numbers reportable as macOS-dev figures; headline NFR-P1 claim stays provisional until bare-metal Linux. Benchmarks need an UNSANITIZED build of the bench binary (sanitizer overhead would invalidate the comparison) — plan a `bench` preset or reuse the unsanitized-target pattern from libshuttle_c. - -- **2026-06-11 (iteration 22 — G6.2 PASS both legs):** Rust leg of the cross-language proof. `tests/ffi/rust/shuttle.rs`: safe wrapper — `Consumer::acquire_read(&mut self) -> Borrowed<'_>`; the payload slice is tied to the Borrowed's lifetime and Drop performs release_read, so use-after-release AND double-acquire are compile errors by construction. `consumer.rs`: verifies 1500 seeded messages in place (no to_vec — zero-copy preserved). `compile_fail.rs`: the use-after-release proof. `tests/cabi_rust_test.cpp` driver, three stages: (1) wrapper+consumer must compile (so stage 2 is meaningful), (2) compile_fail.rs must fail with exactly E0597 (wrong-reason failures rejected), (3) end-to-end C++-producer→Rust-consumer byte-exact over a live channel, rustc invoked at test time via posix_spawnp with rpath to the unsanitized libshuttle_c. Decision: hand-written extern decls over bindgen (logged above). 24/24 ASan+TSan both legs. Next objective: **G6.3** — induced error (open nonexistent segment) surfaces as the correct integer code in all three languages, no exception/panic escaping the ABI. - -- **2026-06-11 (iteration 21 — G6.1 PASS both legs):** Phase 6 C ABI frozen: `include/shuttle/shuttle_c.h` (SHUTTLE_ABI_VERSION 1; SRS §3.1 signatures: create/open/close/unlink, write/read copy path, acquire/commit/acquire/release borrow path, keepalive; SHUTTLE_NONBLOCK flag; error #defines static_asserted against shuttle::Err). `src/shuttle_c.cpp`: every entry point try/catch-wrapped (IF-1), lazy role inference (handle becomes producer/consumer on first use, FR-6), single shared consumer-borrow state so copy-read and acquire-read interoperate; too-small copy-read buffer leaves the message queued. Built as `libshuttle_c` SHARED, unsanitized (decision above). `tests/ffi/py_consumer.py`: cffi ABI-mode binding, blocking acquire_read borrows exposed as memoryview over ffi.buffer (no copy), per-byte verification in place, BorrowedMessage guard raises on use-after-release (minor amendment). `tests/cabi_python_test.cpp`: C++ driver produces 1500 seeded random messages THROUGH the C ABI and posix_spawnp's python3. All four legs re-run on fresh build trees (CMake topology changed): 23/23. Next objective: **G6.2** — C++ producer ↔ Rust consumer byte-exact; Rust wrapper must make use-after-release fail to compile (lifetime on the borrowed slice). Needs rustup/cargo availability check on both legs first. - -- **2026-06-11 (iteration 20 — G5.4 PASS both legs; PHASE 5 COMPLETE):** Kill-while-holding-the-park-mutex, end-to-end through the blocking API. Designing the gate exposed that the planned macOS trylock-loop escape was unsound (cond_timedwait re-acquisition is an unprotectable bare lock) — replaced macOS parking with `os_sync_wait_on_address` per the pre-authorized fallback (decision logged above; seam: park_wait_cursor/park_wake_cursor; spsc park paths now snapshot the peer cursor pre-park and sleep until it changes). `tests/crash_mutex_test.cpp`: crasher writes marker, LOCKS the park mutex, keepalives while holding it; driver SIGKILLs at ~1 s with the victim parked in read(); victim aborts kErrPeerDead at 2.52 s (mac) / 2.63 s (linux), bounded [1.5 s, 8 s]; on linux the driver then locks/unlocks the orphaned mutex through the seam, proving robust recovery left it serviceable. Full regression green over the reworked parking: 22/22 ASan+TSan both legs (one unexplained initial linux failure of trickle+latency — see flake observation in decisions; 11 consecutive passes since). Next objective: **G6.1** — Phase 6: freeze the extern "C" ABI (wrap entry points, no exceptions across the boundary, versioned header for bindgen/cffi), then C++ producer ↔ Python consumer byte-exact over the borrow path (cffi, zero-copy memoryview with release-invalidation per the minor amendment). - -- **2026-06-11 (iteration 19 — G5.3 PASS both legs):** Added `tests/crash_leak_test.cpp`: producer SIGKILLed mid-keepalive; survivor consumer detects death via kErrPeerDead (G5.1 machinery), performs the real-application teardown (close + unlink), and verifies the object is gone; the driver then INDEPENDENTLY re-verifies from a second process — /dev/shm clean on Linux (via the seam's fs view), name not re-openable (kErrNotFound) on both platforms. No library changes needed: NFR-R2 after a crash follows from G5.1's detection plus G1.3's unlink semantics. 21/21 ASan+TSan both legs. Next objective: **G5.4** — kill-while-holding-the-park-mutex, end-to-end through the BLOCKING API on both platforms: Linux survivor recovers via robust path while parked; macOS survivor escapes via the trylock loop + heartbeat staleness. The macOS leg needs a way for the trylock loop to give up — currently it loops forever; wire heartbeat staleness into park_until_* around the lock acquisition (the A3 'never a bare lock' completion). - -- **2026-06-11 (iteration 18 — G5.2 PASS, linux-only gate):** Robust-mutex hardening behind the seam: `mutex_init_pshared` adds PTHREAD_MUTEX_ROBUST on Linux; new `park_mutex_recover_if_needed` absorbs EOWNERDEAD in both `park_mutex_lock` and the timedwait re-acquisition path (App. B #3 order: repair → consistent → continue; repair is a documented NO-OP because the park mutex guards only the advisory waiting flags and bounded-timedwait condvars — all real state is single-writer outside the critical section by design). New `kHasRobustMutex` capability constant keeps platform #ifdefs out of tests. `tests/robust_mutex_test.cpp` (self-skips on macOS): scenario A — holder child SIGKILLed while owning the mutex; survivor's seam lock returns success and the mutex remains fully serviceable (lock/unlock + ETIMEDOUT timedwait); scenario B — deliberately buggy recovery (unlock without consistent) leaves the mutex ENOTRECOVERABLE, proving the permanently-dead-mutex failure mode is real and the test can fail (plan 5b debugging strategy). 20/20 ASan+TSan both legs (mac unaffected by linux-only attr). Next objective: **G5.3** — no shm leak after a crashed run on either platform once the survivor tears down. - -- **2026-06-11 (iteration 17 — G5.1 PASS both legs):** Heartbeat liveness (A3) landed in spsc.hpp: each side bumps its own heartbeat (single-writer plain store, never RMW) on every commit/release, every park iteration, and via the new `keepalive()`; a parked waiter samples the peer's heartbeat per park cycle through a process-local StaleTracker and aborts with new error `kErrPeerDead` when it stops advancing for the threshold (constructor param, default 5 s — deliberately > park_idle_test's 3 s quiet window). DOCUMENTED LIMITATION (also in code): an alive peer that makes zero Shuttle calls is indistinguishable from a dead one — sparse-traffic apps must keepalive() or raise the threshold. Also added the producer zero-copy borrow path (try_acquire_write/acquire_write/commit_write, IF-2/FR-10; reservation strictly process-local per A1, so dying mid-reservation leaves no shared inconsistency — exactly what the crash test exploits), plus a guard rejecting try_write while an acquire is outstanding (would have double-reserved the same span). `tests/crash_heartbeat_test.cpp`: crasher writes marker, acquires 4 KiB, keepalives at 50 ms; driver SIGKILLs it at 1.0 s; victim (1 s threshold) read aborts kErrPeerDead at 2.08 s, bounded [1 s, 6 s], and try_read afterward is WouldBlock (no phantom data). 19/19 ASan+TSan both legs. NOTE: G4.3 and G5.1 share one commit — a model-availability outage blocked the intermediate git commit; both gates verified independently as recorded above. Next objective: **G5.2** — Linux robust mutex: PTHREAD_MUTEX_ROBUST init in the container, EOWNERDEAD → repair → pthread_mutex_consistent → unlock (in that order), kill-while-holding-mutex; plus the deliberately-buggy-recovery check that the test can detect a permanently dead mutex. - -- **2026-06-11 (iteration 16 — G4.3 PASS both legs; PHASE 4 COMPLETE):** Added instance-local `locks_taken()` counters at every park_mutex_lock site in spsc.hpp (no segment/ABI impact). `tests/park_latency_test.cpp`: (1) latency — producer stamps CLOCK_MONOTONIC into each payload pre-write, paced 200 µs so the consumer is parked at every commit; p99 asserted < 1 ms; measured p50/p99/max: 7/39/664 µs mac ASan, 85.2/198.3/11368 µs linux container ASan (virtualized, not headline figures); (2) hotpath — dual-thread, 64 MiB capacity, consumer spins try_read (never parks): both handles end at exactly 0 lock acquisitions. 18/18 under ASan+TSan both legs. Next objective: **G5.1** — heartbeat liveness (A3, both platforms): bump on activity + while parked, staleness verdict at timedwait timeouts, kErrPeerDead abort; SIGKILL-producer-mid-reservation crash test. Design note: an idle-but-alive peer that makes NO shuttle calls is indistinguishable from dead — provide keepalive() and document; threshold is process-local policy (constructor param), default 5 s (must stay > park_idle_test's 3 s quiet window). - -- **2026-06-11 (iteration 15 — G4.2 PASS both legs):** `tests/trickle_test.cpp`: producer sends 100k small messages at random 0–80 µs intervals so the consumer parks for nearly every one. Lost-wakeup detection is sharp: the 100 ms timedwait backstop would MASK a lost wake as a ~100 ms stall, so every read is timed and >5 reads over 50 ms fails the run — actual result 0 slow reads, max wait 7.4 ms (mac). "No extras" verified: exactly N byte-exact FIFO messages then WouldBlock. Note: linux-container leg takes ~100 s (coarser usleep granularity stretches the trickle), within deadlines. 16/16 under ASan+TSan, both legs. Next objective: **G4.3** — µs-scale p99 wake latency (cross-process CLOCK_MONOTONIC timestamps in payload per minor amendment) + prove the hot path takes zero mutex acquisitions when the peer isn't parked (lock counting). - -- **2026-06-10 (iteration 14 — G4.1 PASS both legs):** Phase 4 parking-lot wake landed. `platform.hpp`: `park_mutex_lock` seam — macOS is a trylock loop with 100 µs sleeps (never a bare lock, per A3; heartbeat check slots in Phase 5), Linux plain lock (robust recovery slots in Phase 5b). `spsc.hpp`: blocking write/read now spin briefly (256 iterations) then park; park decision is the A4 Dekker protocol — waiter stores its waiting flag, seq_cst fence, re-checks the predicate; signaler publishes the cursor, seq_cst fence, loads the flag (the PARKING PROTOCOL comment block carries the full argument: in the seq_cst total order one of the two must see the other). Lost-wakeup guard: predicate re-checked under the park mutex before every `cond_timedwait_rel` (always deadlined, 100 ms — A3). Wakes fire from commit (not_empty), release AND the A→B handoff (not_full — the handoff frees space too). Peer-not-parked cost: one fence + one relaxed load, no mutex. try_* paths and all cursor orderings untouched; modifying the gated blocking paths IS Phase 4's planned variable (plan: "replace the busy-poll"), and all Phase 3 gates re-ran green over the new implementation. `tests/park_idle_test.cpp`: idler child self-measures rusage across a 3 s empty-channel blocking read — 1.4 ms CPU (0.05%), wake message byte-exact. 15/15 under ASan+TSan, both legs. Next objective: **G4.2** — trickle stress: one message every random interval, ≥100k messages (will take a while at true trickle pace — size intervals so the run fits ctest TIMEOUT), no lost/extra wakeups. - -- **2026-06-10 (iteration 13 — G3.3 PASS both legs; PHASE 3 COMPLETE):** Added `tests/spsc_wrap_test.cpp`: 16 KiB channel with 2–6 KiB payloads so the early-wrap commit (P2) and consumer handoff (C2) fire every ~3 messages. Two-process: 200k msgs, 57,822 wraps, byte-exact FIFO, drained at end; producer counts its own write-cursor backward moves and FAILS below 25k wraps (falsifiable). Dual-thread A2 config (50k msgs) TSan-clean on both legs. No library changes — the spsc.hpp ordering survived the targeted hammering of its hotspot. **Phase 3 done in 3 iterations with zero ordering bugs found** — the Phase 2 property-test investment did its job; the lock-free protocol is now trusted. Next objective: **G4.1** — Phase 4 parking-lot wake: pshared mutex + condvars wired into the blocking paths (timedwait + heartbeat per A3 from day one; seq_cst parking protocol per A4), idle blocked peer at ~0% CPU. Remember: any bug from here is BY CONSTRUCTION a wake bug, not an ordering bug. - -- **2026-06-10 (iteration 12 — G3.2 PASS both legs):** Added `tests/spsc_asym_test.cpp`: two two-process scenarios over a deliberately small 256 KiB channel — (A) full-speed producer vs consumer throttled 100 µs/msg (buffer saturates, producer spins on FULL) and (B) the reverse (consumer spins on EMPTY). 5000 random-sized messages each way, byte-exact FIFO. Falsifiability: the fast side counts messages where try_* returned kErrWouldBlock at least once and fails the run if fewer than 500 — the spin paths must demonstrably engage. No library changes. 12/12 under ASan+TSan, both legs. Next objective: **G3.3** — targeted A→B-handoff stress: capacity sized so wraps happen every few messages, byte-exact under the A2 TSan configuration (dual-thread mode) as well as two-process. - -- **2026-06-10 (iteration 11 — G3.1 PASS both legs):** Phase 3 lock-free data path landed in `include/shuttle/spsc.hpp`: Producer/Consumer over the shared header's atomic cursors, busy-poll on empty/full (cpu_relax + periodic yield via the platform seam), copy-write + zero-copy-borrow read. The normative memory-ordering contract is the file-top comment block (P1 payload publish via write release/acquire; P2 watermark relaxed-store sequenced before write release; C1 free publish via read release/acquire; C2 handoff = single owned-variable store, no torn state; staleness argument: re-loads only under-estimate). Consumer parse validates length against max_payload and available run (NFR-S2) → kErrCorrupt. New error codes kErrMsgTooLarge, kErrWouldBlock. `tests/spsc_stress_test.cpp`: driver/producer/consumer roles via posix_spawn (40k seeded random messages ≈1.25 GiB, byte-exact FIFO, drained-at-end check) + `threads` mode (same producer_loop/consumer_loop functions, two threads, real MAP_SHARED segment, 8k msgs) registered as shuttle_spsc_threads_test — the TSan-attaching config per A2. All 11 tests pass: ASan both legs; TSan both legs clean (two-proc 34 s mac / 23 s linux under TSan; threads 7 s / 4 s). ctest TIMEOUT 300 for the stress pair. Next objective: **G3.2** — asymmetric-speed stress both directions (producer-fast forces backpressure-by-spin; consumer-fast forces empty-spin), byte-exact both ways. - -- **2026-06-10 (iteration 10 — G2.3 PASS both legs; PHASE 2 COMPLETE):** Added `tests/bipbuffer_oversize_test.cpp`: a framed unit > capacity is refused immediately (both write_msg and raw reserve), with a cursor/size snapshot proving ZERO state mutation, tested from empty, partially-filled, and wrapped states; live data drains byte-exact afterward; the exact max_payload (= cap − 8) boundary still fits. One test-authoring bug caught and fixed during the iteration: the wrapped-state section initially reused the previous section's buffer, so its early-wrap setup started from r=w=300 and was correctly refused by the strict `read > n` rule — sections now each use a fresh buffer (test construction fix; the gate's oversize assertions were never touched). No library changes. 9/9 under ASan+TSan, both legs. **Phase 2 done — the BipBuffer arithmetic is fully trusted.** Next objective: **G3.1** — Phase 3, THE HARD ONE: promote cursors to atomics in the shared header, lock-free release/acquire protocol, busy-poll empty/full, two-process ≥1 GB byte-exact FIFO stress; TSan-clean applies to the single-process dual-thread configuration per amendment A2; write the happens-before argument as inline comments for every shared atomic. Budget multiple iterations; the A→B handoff ordering is the expected trouble spot. - -- **2026-06-10 (iteration 9 — G2.2 PASS both legs):** Added `tests/bipbuffer_edge_test.cpp`, deterministic scripts with hand-computed cursor expectations for the three named edges: (1) exact-fill after A — write lands exactly on the physical end (`write == cap`), next write early-wraps rather than failing; (2) forced early wrap to B — wrapped cursors verified (w=500/m=800/r=600), payload pointer-identity at offset 0 after handoff, PLUS the two strict-inequality refusals (linear wrap with unit == read refused; wrapped append with unit == read−write refused, unit one less accepted) that guard wrapped-full vs linear-empty aliasing; (3) max-size payload — single message occupying the whole buffer, nothing more accepted while full, clean post-drain wrap, and the SRS 2× shape with two max messages in flight simultaneously. 8/8 under ASan+TSan, both legs. No library changes needed. Next objective: **G2.3** — a write of max_payload + 1 (logically, > usable capacity) fails fast rather than looping or wrapping incorrectly. - -- **2026-06-10 (iteration 8 — G2.1 PASS both legs):** Phase 2 BipBuffer landed in `include/shuttle/bipbuffer.hpp`: pure single-threaded logic, amendment-A1 cursor model (read/write/watermark absolute offsets, single-writer each; regions A/B derived only; reserve state producer-private). Key encoded rules: strict `write < read` in all wrapped-space checks (wrapped-full must never alias linear-empty `write == read`), whole-unit early wrap to offset 0, A→B handoff = consumer storing `read = 0` on observing `read == watermark && write < read`, explicit little-endian u64 framing. `kFrameHeader` moved from header.hpp into bipbuffer.hpp (constant relocation only; header.hpp now includes bipbuffer.hpp). `tests/bipbuffer_test.cpp`: model-based property test, payload bytes regenerated from msg index (nothing stored), 100k pairs roomy (64 KB cap, 1496 wraps) + 100k pairs tight (4 KB cap, 19267 wraps), invariants (cursors ≤ cap; wrapped ⇒ read ≤ watermark; derived size == modeled in-flight bytes) after every op, byte-exact FIFO. 7/7 under ASan+TSan, both legs. Next objective: **G2.2** — the three named edge cases: payload exactly filling the space after A; forced early wrap to B; max-size payload. - -- **2026-06-10 (iteration 7 — G1.3 PASS both legs; PHASE 1 COMPLETE):** Added `shm_object_exists_fs` to the platform seam (Linux: stat of /dev/shm/; macOS: -1 = no filesystem view, callers fall back to behavioral proof) and `tests/leak_test.cpp`: live object visible in /dev/shm (ground truth the check can fail), survives close() per FR-5, gone from /dev/shm after unlink, not re-openable (kErrNotFound) on both platforms, double unlink reports kErrNotFound. 6/6 tests pass under ASan and TSan on both legs. Phase 1 done: lifecycle, header, validation, leak hygiene all gated. Next objective: **G2.1** — Phase 2 BipBuffer logic, single-threaded over a heap buffer, NO shared memory: reserve/commit/read_block/release with the amendment-A1 read/write/watermark invariant (regions A/B strictly derived, never stored), early-wrap rule, 8-byte framing; gate = byte-exact FIFO over ≥100k random write/read pairs with invariants asserted after every op. This is the phase for heavy property testing — invest in the fuzz harness, it is what de-risks Phase 3. - -- **2026-06-10 (iteration 6 — G1.2 PASS both legs):** Added `tests/capacity_test.cpp` (single-process; FR-4 needs no second process): capacities of maxp+7, maxp, and 1 against a 64 KB max_payload all fail with the distinct kErrCapacityTooSmall; a failed create leaves no shm object behind (open afterward → kErrNotFound); the exact boundary capacity == max_payload + kFrameHeader succeeds, then closes and unlinks cleanly. No library-code changes needed — the validation landed with G1.1. 5/5 tests pass under ASan and TSan on both legs. Next objective: **G1.3** — leak check (NFR-R2): after create→close→unlink, /dev/shm is clean on Linux and the name cannot be re-opened on either platform. - -- **2026-06-10 (iteration 5 — G1.1 PASS both legs):** Phase 1 lifecycle landed: `include/shuttle/header.hpp` (full ChannelHeader per amendment A1 — read/write/watermark single-writer cursors, parking flags, heartbeats, pshared primitives; every hot atomic on its own 128-byte line, offsets static_asserted, layout change = version bump), `include/shuttle/shuttle.hpp` (Err codes + create/open/close/unlink), `src/shuttle.cpp` (O_CREAT|O_EXCL + one-shot ftruncate + mmap + init, release-publish of init_state; opener acquire-spins with 5 s deadline → kErrInitTimeout; magic then version then geometry validation with distinct errors), `shm_name_ok` in the platform seam (30-char macOS cap), `tools/inspect.cpp` (shuttle_inspect header-dump CLI), `tests/proc_util.hpp` (posix_spawn child helper per fork-ban amendment), `tests/lifecycle_test.cpp` (the G1.1 test). One real bug found by the mac leg: geometry equality check vs macOS page-size-rounded `st_size` (decisions log). All 4 tests pass under ASan and TSan on both legs. Next objective: **G1.2** — `create` with `capacity < max_payload + 8` fails with the FR-4 error code, not a crash (validation already implemented in create(); needs its gate test). - -- **2026-06-10 (iterations 2–4 — G0.2, G0.3, G0.4 all PASS; PHASE 0 COMPLETE):** - - **G0.2:** TSan legs pass. Container hiccup: Docker's default seccomp blocks `personality(2)`, so `setarch -R` failed; fixed with `--security-opt seccomp=unconfined` on the TSan target only (see decisions log — Docker Desktop's kernel has `mmap_rnd_bits=18` so TSan happens to work without setarch, but we keep the workaround functional for stock-Ubuntu kernels). - - **G0.3:** `shuttle_shm_smoke` maps/touches/unmaps 4 KB and 128 MB segments on both legs. Negative control verified: the same binary SIGBUSes in a container with the default 64 MB /dev/shm, proving the test actually guards the `--shm-size` gotcha. - - **G0.4:** `shuttle_pshared_smoke` — driver posix_spawn's the binary as `waiter`/`signaler` (fork-without-exec forbidden per amendment); waiter parks on a pshared condvar in shm, signaler wakes it; per-run nonce; every wait deadlined (timeout=failure; ctest TIMEOUT 60 as backstop). **PASS on macOS 26.5 natively** — the highest-risk Phase 0 unknown resolved positively; pshared mutex+condvar work cross-process on this host, no `os_sync_wait_on_address` fallback needed. Also PASS in container, and clean under TSan on both legs. `platform.hpp` gained its first real seam functions: `mutex_init_pshared`, `cond_init_pshared_monotonic` (CLOCK_MONOTONIC on Linux), `cond_timedwait_rel` (timedwait vs `pthread_cond_timedwait_relative_np`), `monotonic_ns`. - - Next objective: **G1.1** — Phase 1 segment lifecycle: header struct with static_asserts, offsets-not-pointers discipline, create/open/close/unlink, magic+version validation, single-init publication, short-name + one-shot-ftruncate enforcement in the platform seam, `shuttle_inspect` CLI. - -- **2026-06-10 (iteration 1 — G0.1 PASS both legs):** Docker now installed and verified (pull + native arm64 run of ubuntu:24.04). Built the Phase 0 skeleton: `CMakeLists.txt` (C++17, three targets `shuttle_core`/`shuttle_tests`/`shuttle_bench`, core free of test/bench deps, `SHUTTLE_SAN` sanitizer config), `include/shuttle/platform.hpp` (the single platform seam, stub), empty test + bench stub, `docker/Dockerfile` (ubuntu:24.04 glibc arm64), `Makefile` with one-command legs `test-mac`/`test-linux`/`tsan-mac`/`tsan-linux` (tsan targets wired but NOT yet verified — that is G0.2). Linux leg passed first try. Mac leg initially HUNG: ASan runtime spin pre-`main` (see decisions log) — fixed by CLT 26.5 update, then `ctest` 1/1 Passed under ASan+UBSan. Both legs re-verified on the final tree. Next objective: **G0.2** — TSan preset links and runs the empty test on both legs (`make tsan-mac`, `make tsan-linux`; linux leg already wired with `setarch -R`). - -- **2026-06-09 (iteration zero):** Read all docs. Verified environment: Xcode CLT and cmake OK; **Docker is not installed on this host** (not merely not running). Standing orders mandate STOP-and-tell-the-user when Docker is unavailable — the Linux leg can never be faked or skipped, and G0.1 cannot pass on both legs without it. Initialized git, created this ledger, ended the iteration with no Phase 0 code written (deliberate: every G0.x gate needs the Linux leg, so writing the CMake skeleton before the container runtime exists risks verifying against an assumption). **User action required:** install and start a Docker runtime (Docker Desktop for Mac, or colima/OrbStack — anything providing a `docker` CLI that runs linux/arm64 containers), able to `docker pull ubuntu:24.04` (glibc arm64, never Alpine) and `docker run --shm-size=512m`. Next iteration: re-verify Docker, pull base image, then begin Phase 0 work toward G0.1 (CMake skeleton with shuttle_core/shuttle_tests/shuttle_bench targets, platform.hpp seam, ASan+UBSan and separate TSan presets, Dockerfile, `make test-mac` / `make test-linux` single commands). diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..6e7ba8a --- /dev/null +++ b/docs/API.md @@ -0,0 +1,360 @@ +# Shuttle C ABI Reference + +The frozen `extern "C"` surface declared in `include/shuttle/shuttle_c.h`. This +header is the single source of truth for foreign-language bindings; it is pure C +(fixed-width / standard types, no C++). No exception ever crosses the boundary — +every failure is an integer code. `SHUTTLE_ABI_VERSION` is `1`. + +Two conventions for reporting failure: + +- Functions returning `shuttle_channel*` report the code through an out-param + `int* err` and return `NULL` on failure. +- Functions returning `int` return `SHUTTLE_OK` (0) or a negative error code. +- `shuttle_read` returns a `long`: the non-negative payload length on success, + or a negative error code. + +The ten functions `shuttle_create` .. `shuttle_keepalive` are frozen v1. +`shuttle_create_ex` is an additive v1.1 symbol (new symbol only, no existing +signature or semantic touched), so the ABI version stays `1`. + +--- + +## Functions + +### shuttle_create + +```c +shuttle_channel* shuttle_create(const char* name, size_t capacity_bytes, + size_t max_payload_bytes, int* err); +``` + +Create and initialize a new channel: `shm_open(O_CREAT|O_EXCL)`, one-shot +`ftruncate`, `mmap`, header init, then a release-store that publishes readiness. +Owner-only permissions (`0600`). The caller of `create` is neither producer nor +consumer yet — the role is bound lazily on first use of the corresponding path. + +- Blocking: no. Returns immediately. +- Segment size mapped: `sizeof(ChannelHeader) + capacity_bytes`. +- Errors (via `*err`): `INVALID_ARGS` (`name` NULL, `name[0] != '/'`, + `capacity_bytes == 0`, or `max_payload_bytes == 0`), `NAME_TOO_LONG`, + `CAPACITY_TOO_SMALL` (`capacity_bytes < max_payload_bytes + 8`), `EXISTS` + (name already present), `SYS` (`ftruncate`/`mmap`/other syscall failure). + +### shuttle_create_ex + +```c +#define SHUTTLE_CREATE_HUGEPAGES 0x1 +shuttle_channel* shuttle_create_ex(const char* name, size_t capacity_bytes, + size_t max_payload_bytes, + uint32_t create_flags, int* err); +``` + +As `shuttle_create`, plus opt-in create-time behavior selected by +`create_flags`. `create_flags` is a **separate namespace** from the per-op +`flags` argument used by the read/write calls (`SHUTTLE_NONBLOCK`); the two must +not be mixed. Unknown flag bits are masked off and ignored (never persisted). + +- `SHUTTLE_CREATE_HUGEPAGES`: advise transparent huge pages for the mapping via + `madvise(MADV_HUGEPAGE)` on Linux. Purely advisory — it takes effect only + where the kernel THP shmem policy permits, and a kernel that disallows it + returns a harmless `EINVAL` that is dropped. No-op on macOS and on + unsupported kernels; never a correctness dependency. The chosen flag is + recorded in the segment header, so that `shuttle_open` re-advises the opener's + independent mapping automatically. +- Errors: identical set to `shuttle_create`. Passing only unknown bits is not + an error; they are silently masked. + +### shuttle_open + +```c +shuttle_channel* shuttle_open(const char* name, int* err); +``` + +Attach to an existing channel without re-initializing it. Waits (deadlined, 5 s) +for the creator's readiness publication, then validates magic, version, and +header geometry before trusting any field. If the creator opted into huge pages, +the opener's mapping is advised too. + +- Blocking: bounded. Spins up to a 5 s deadline waiting for creator init. +- Errors (via `*err`): `INVALID_ARGS` (`name` NULL or `name[0] != '/'`), + `NOT_FOUND` (no such segment), `CORRUPT` (`fstat` failure, segment smaller + than a header, or header geometry fails validation), `SYS` (`mmap` failure), + `INIT_TIMEOUT` (creator never published readiness within 5 s), `BAD_MAGIC`, + `BAD_VERSION`. + +### shuttle_close + +```c +void shuttle_close(shuttle_channel* ch); +``` + +Unmap the segment and free the local handle (and any producer/consumer state it +created). The named shm object survives — use `shuttle_unlink` to remove it. +NULL-safe. Never fails. + +### shuttle_unlink + +```c +int shuttle_unlink(const char* name); +``` + +Remove the named shm object. Existing mappings remain valid until closed +(POSIX unlink semantics). + +- Returns: `SHUTTLE_OK`, `INVALID_ARGS` (`name` NULL or `name[0] != '/'`), + `NOT_FOUND` (no such object), `SYS` (other failure). + +### shuttle_write + +```c +int shuttle_write(shuttle_channel* ch, const void* data, size_t len, int flags); +``` + +Copy path: frame and enqueue a single message of `len` bytes. Binds the handle +as producer on first use. + +- Blocking (`flags == 0`): brief adaptive spin, then park until space. Returns + `SHUTTLE_OK` or `PEER_DEAD`. +- Non-blocking (`flags & SHUTTLE_NONBLOCK`): try-semantics; returns + `WOULD_BLOCK` instead of parking. +- Errors: `INVALID_ARGS` (`ch` NULL, or `data` NULL with `len != 0`), + `MSG_TOO_LARGE` (`len > max_payload`; checked fail-fast, never parks), + `WOULD_BLOCK` (non-blocking, no contiguous space now), `PEER_DEAD` (blocked + wait aborted — consumer heartbeat went stale), `SYS`. + +### shuttle_read + +```c +long shuttle_read(shuttle_channel* ch, void* out, size_t cap, int flags); +``` + +Copy path: dequeue the next message into `out`, up to `cap` bytes. Binds the +handle as consumer on first use. + +- Returns the payload length (`>= 0`) on success. +- Blocking / non-blocking as for `shuttle_write`. +- If the waiting message is larger than `cap`, returns `MSG_TOO_LARGE` **and the + message stays queued** (not consumed) — retry with a larger buffer. +- Errors: `INVALID_ARGS` (`ch` NULL, or `out` NULL with `cap != 0`), + `MSG_TOO_LARGE`, `WOULD_BLOCK` (non-blocking, empty), `PEER_DEAD` (blocked + wait aborted — producer heartbeat went stale), `CORRUPT` (framed length in + the segment is impossible — never hands out an out-of-bounds span), `SYS`. + +### shuttle_acquire_write + +```c +int shuttle_acquire_write(shuttle_channel* ch, void** ptr, size_t len, int flags); +``` + +Zero-copy borrow, producer side: reserve a **contiguous** writable span of `len` +bytes and return a pointer to it in `*ptr`. Nothing is published until +`shuttle_commit_write`. At most one outstanding reservation per handle. The +reservation is process-local: a producer that dies mid-reservation leaves no +shared-state inconsistency. + +- Blocking / non-blocking as for `shuttle_write`. +- Errors: `INVALID_ARGS` (`ch` or `ptr` NULL, or a reservation is already + outstanding), `MSG_TOO_LARGE` (`len > max_payload`), `WOULD_BLOCK` + (non-blocking, no contiguous span now), `PEER_DEAD`, `SYS`. + +### shuttle_commit_write + +```c +int shuttle_commit_write(shuttle_channel* ch, size_t actual_len); +``` + +Publish `actual_len` bytes (`<= len` reserved) of the outstanding reservation. +This is the release edge that makes the payload visible to the consumer. + +- Errors: `INVALID_ARGS` (`ch` NULL, no producer role, no active reservation, or + `actual_len > len` reserved), `SYS`. + +### shuttle_acquire_read + +```c +int shuttle_acquire_read(shuttle_channel* ch, const void** ptr, size_t* len, int flags); +``` + +Zero-copy borrow, consumer side: borrow the next message in place. `*ptr` points +into the shared segment; `*len` is the payload length. At most one outstanding +borrow per handle; must be paired with `shuttle_release_read`. + +- Blocking / non-blocking as for `shuttle_read`. +- The borrowed payload is guaranteed contiguous (see Zero-copy borrow rules). +- Errors: `INVALID_ARGS` (`ch`, `ptr`, or `len` NULL), `WOULD_BLOCK` + (non-blocking, empty), `PEER_DEAD`, `CORRUPT`, `SYS`. + +### shuttle_release_read + +```c +int shuttle_release_read(shuttle_channel* ch); +``` + +Release the outstanding read borrow, freeing its bytes for producer reuse. This +is the release edge; the borrowed pointer is invalid afterward. + +- Errors: `INVALID_ARGS` (`ch` NULL, no consumer role, or no active borrow), + `SYS`. + +### shuttle_keepalive + +```c +void shuttle_keepalive(shuttle_channel* ch); +``` + +Bump this side's heartbeat without transferring data. Sparse-traffic peers must +call this (or raise the staleness threshold) so the other side does not declare +them dead. NULL-safe. Bumps whichever roles (producer/consumer) the handle has +already taken. Never fails. + +--- + +## Error codes + +| Value | Name | Meaning / when it occurs | +|-------|-------------------------------|--------------------------| +| 0 | `SHUTTLE_OK` | Success. | +| -1 | `SHUTTLE_ERR_INVALID_ARGS` | NULL handle/pointer, malformed name, zero size, misuse (double reservation/borrow, wrong role). | +| -2 | `SHUTTLE_ERR_NAME_TOO_LONG` | Name exceeds the platform shm-name limit (macOS 30, Linux 254). | +| -3 | `SHUTTLE_ERR_EXISTS` | `create`: a segment with that name already exists. | +| -4 | `SHUTTLE_ERR_NOT_FOUND` | `open`/`unlink`: no such segment. | +| -5 | `SHUTTLE_ERR_SYS` | Unexpected syscall failure (inspect `errno`) or a caught exception. | +| -6 | `SHUTTLE_ERR_BAD_MAGIC` | `open`: segment magic word mismatch. | +| -7 | `SHUTTLE_ERR_BAD_VERSION` | `open`: layout version mismatch (distinct from magic). | +| -8 | `SHUTTLE_ERR_CAPACITY_TOO_SMALL` | `create`: `capacity_bytes < max_payload_bytes + 8`. | +| -9 | `SHUTTLE_ERR_INIT_TIMEOUT` | `open`: creator never published readiness within 5 s. | +| -10 | `SHUTTLE_ERR_CORRUPT` | Header geometry fails validation, or a framed length in the data region is impossible. | +| -11 | `SHUTTLE_ERR_MSG_TOO_LARGE` | Write payload `> max_payload`; or copy-read buffer smaller than the queued message (message stays queued). | +| -12 | `SHUTTLE_ERR_WOULD_BLOCK` | Non-blocking op cannot proceed right now (full on write, empty on read). | +| -13 | `SHUTTLE_ERR_PEER_DEAD` | A blocking wait aborted because the peer's heartbeat went stale. | + +The C `#define`s are `static_assert`ed equal to the C++ `shuttle::Err` enum in +the implementation, so the two can never drift. + +--- + +## Channel lifecycle + +- **create / open**: exactly one creator (`shuttle_create[_ex]`, `O_CREAT|O_EXCL`) + and any number of openers (`shuttle_open`). The channel is SPSC: one producer, + one consumer. Roles are bound lazily — a handle becomes a producer on its + first write/acquire-write, a consumer on its first read/acquire-read. +- **close**: unmaps and frees the local handle only; the named object persists. +- **unlink**: removes the named object; live mappings stay valid until closed. +- **Name constraints**: must begin with `/`, contain at least one further + character (length `>= 2`), and not exceed the platform limit — **30 chars on + macOS** (`PSHMNAMLEN` is 31 including the `/`; 30 avoids the documented + off-by-one), **254 chars on Linux**. Over-limit names yield `NAME_TOO_LONG`. +- **Sizing rule**: `capacity_bytes >= max_payload_bytes + 8`. The `+8` is the + frame header (an 8-byte little-endian length prefix on every message). This + makes a permanently-unsatisfiable write impossible by construction, so + blocking backpressure can never park the producer forever. Violation yields + `CAPACITY_TOO_SMALL`. +- **One-shot sizing**: the segment is `ftruncate`d exactly once, at creation, on + both platforms (macOS forbids re-truncating an shm object). It never grows. + +--- + +## Zero-copy borrow rules + +- **Pointer validity window**: a pointer from `shuttle_acquire_read` (or `*ptr` + from `shuttle_acquire_write`) is valid only until the matching release + (`shuttle_release_read` / `shuttle_commit_write`). After release the bytes may + be reused by the peer; dereferencing the stale pointer is undefined. +- **Mandatory release**: every acquire must be paired with exactly one release. + At most one borrow/reservation may be outstanding per handle; a second acquire + before releasing the first returns `INVALID_ARGS`. +- **Contiguity guarantee**: the ring is a **BipBuffer** — reservations are + whole-unit, so a payload is never split across the physical wrap. A borrowed or + reserved span is always one contiguous run of bytes, safe to expose directly as + a slice / `memoryview` / NumPy array with no copy and no stitching. + +--- + +## Segment layout + +`sizeof(ChannelHeader)` (a cache-line multiple) of control header, followed by +the data region at byte offset `data_offset`. Everything in the segment is +fixed-width and referenced by **byte offset from the segment base**, never by +pointer — each process maps at a different address. + +Header, in order: + +1. **Cold identity block** (written once by the creator, immutable after + `init_state` is published): `magic`, `version`, `flags`, `data_offset`, + `data_capacity`, `max_payload`, then the `init_state` publication atomic. +2. **BipBuffer cursors**, each on its own 128-byte line, each strictly + single-writer: `write` (producer), `watermark` (producer), `read` (consumer). + If `write >= read`, valid data is `[read, write)`; else `[read, watermark)` + then `[0, write)`. +3. **Park flags**: `producer_waiting`, `consumer_waiting`. +4. **Heartbeats**: `producer_heartbeat`, `consumer_heartbeat`. +5. **Park/wake primitives** (off the hot path): a pshared mutex and two condvars. + +**Flags-bits contract**: the creator writes the entire `flags` word **once**, in +the cold identity block, before the release-store that publishes `init_state`; +it is immutable thereafter. Openers must **ignore unknown bits** — `flags` is an +additive extension point, so new bits carry no version bump (an old opener +simply does not act on a bit it does not recognize). `SHUTTLE_CREATE_HUGEPAGES` +is recorded here as bit `0x1`. + +--- + +## Liveness + +Each side maintains a monotonically increasing heartbeat, bumped on every +successful operation, on every park iteration, and via `shuttle_keepalive`. A +**blocked** waiter samples the peer's heartbeat at each park timeout (100 ms +intervals); if it has not advanced within the staleness threshold (default 5 s, +process-local policy — not stored in the segment), the wait aborts with +`SHUTTLE_ERR_PEER_DEAD` instead of blocking forever. + +**Documented caveat**: a peer that is alive but makes no Shuttle calls at all is +indistinguishable from a dead one. Applications with sparse traffic must call +`shuttle_keepalive` periodically (or raise the staleness threshold) to avoid a +spurious `PEER_DEAD`. The heartbeat only fires on the *blocking* paths; +non-blocking callers never see `PEER_DEAD`. + +--- + +## Language bindings + +### Python (cffi, ABI mode) + +Open the library with `ffi.dlopen`, declare the subset you use with `ffi.cdef`, +then drive the borrow path. The payload is exposed **zero-copy** as a +`memoryview` over `ffi.buffer` wrapping the borrowed pointer — no `bytes()` copy +is ever taken; verification indexes the view in place: + +```python +mv = memoryview(ffi.buffer(ptr, length)) # no copy +``` + +Because Python cannot enforce the borrow lifetime, the reference binding wraps +the view in a guard object that **invalidates on `release_read`**: the view is +`.release()`d and further access raises `RuntimeError`. Shuttle cannot prevent a +use-after-release, but it can refuse to serve a stale one. + +For numeric payloads, wrap the same memoryview with NumPy — still zero-copy, no +intermediate buffer: + +```python +arr = np.frombuffer(mv, dtype=np.uint8) # a view over the borrow, no copy +``` + +Release before (or without) touching `arr` afterward; it aliases segment memory +that the producer may overwrite once released. + +### Rust + +The reference wrapper (`tests/ffi/rust/shuttle.rs`) hand-writes the `extern "C"` +declarations against the frozen header and encodes the borrow lifetime in the +type system. `acquire_read(&mut self)` borrows the `Consumer` mutably and returns +`Borrowed<'_>`; the slice from `as_slice()` is tied to the `Borrowed`'s lifetime, +and `Borrowed`'s `Drop` performs `shuttle_release_read`. Consequences enforced +**at compile time**: + +- A slice cannot outlive the release — use-after-release is a borrow-check error + (`E0597`), not a runtime fault. +- No second acquire while a borrow is outstanding — the `Consumer` stays mutably + borrowed until the `Borrowed` drops. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md new file mode 100644 index 0000000..26dd006 --- /dev/null +++ b/docs/ROADMAP.md @@ -0,0 +1,61 @@ +# Shuttle Roadmap + +Honest triage of proposed upgrade directions. No dates, no promises — just +direction and rationale. Entries are grouped by how settled they are. + +## Already in v1 + +Shipped and tested; listed here because they are frequently proposed as +"future work" when they already exist. + +- **Lock-free acquire/release SPSC data path.** Single-writer atomic cursors, no + mutex on the hot path; the park mutex guards only the park/wake handshake. + See `include/shuttle/spsc.hpp` (memory-ordering contract in the header + comment). +- **Zero-copy Python memoryview.** The borrowed payload is a `memoryview` over + the shared segment — no copy — with a guard that invalidates on release. + See `tests/ffi/py_consumer.py`. +- **Frozen C ABI with `#[repr(C)]` layout contracts.** A stable `extern "C"` + surface consumed by hand-written Rust externs whose types encode the borrow + lifetime. See `tests/ffi/rust/shuttle.rs` and `include/shuttle/shuttle_c.h`. +- **Read-only segment introspection tool.** Dumps the header (cursors, flags, + heartbeats, geometry) without validating or mutating. See `tools/inspect.cpp`. + +## Landing in this change + +- **Opt-in transparent huge pages (THP).** `shuttle_create_ex` with + `SHUTTLE_CREATE_HUGEPAGES` advises the mapping via `madvise(MADV_HUGEPAGE)` on + Linux. Advisory only: a no-op on macOS and where the kernel THP policy + disallows it, never a correctness dependency. The flag is recorded in the + segment header so the opener advises its own mapping too. Unknown create-flag + bits are masked. + +## v1.x candidates + +Plausible next steps; each carries a caveat that keeps it out of v1. + +- **Explicit hugetlbfs pages.** Reserved 2 MB / 1 GB pages (optionally + `MADV_COLLAPSE`) for guaranteed large-page backing rather than THP's advisory + best-effort. Needs dedicated hardware with reserved huge pages to test — CI + shared runners cannot provide it. +- **Stats counters in the shared header.** Message/byte counters surfaced + through the inspect tool. Touches the frozen segment layout, so it needs + versioning care (the layout freeze is an ABI contract). +- **Configurable backpressure policies.** Lossy drop/overwrite modes for callers + that prefer freshness over completeness. Must be **strictly opt-in**: + never-drops is a v1 guarantee, so any lossy mode is an explicit, + separately-selected policy — never a default or a silent fallback. +- **Bare-metal Linux benchmark run.** Convert the provisional headline latency + claim into a final one by measuring on controlled hardware (shared CI runners + produce arbitrary numbers and cannot settle a percentile claim). + +## Exploratory / v2 + +Larger directions that would reshape the design; captured for direction, not +committed. + +- **CUDA IPC / GPU-direct interop.** Let GPU-bound payloads move without a + CPU-RAM round-trip, for pipelines where both ends already live on the device. +- **Windows named shared memory.** A `CreateFileMapping`-based platform seam + alongside the current POSIX one. v1 is Linux + macOS only; Windows is a whole + new backend, not a port. diff --git a/docs/SHUTTLE_AGENT_PROMPT.md b/docs/SHUTTLE_AGENT_PROMPT.md deleted file mode 100644 index 58ee465..0000000 --- a/docs/SHUTTLE_AGENT_PROMPT.md +++ /dev/null @@ -1,203 +0,0 @@ -# Shuttle — Autonomous Build Directive (macOS host edition) - -You are implementing **Shuttle**, a zero-copy shared-memory SPSC IPC framework in C++17. -This file is your standing orders. It is designed to be re-read at the start of **every** -loop iteration, so each iteration is idempotent and self-orienting. - -## Host platform - -The host is a **Mac (Apple Silicon, M-series) running macOS**. This is the intended -two-platform dev box, so BOTH gate legs run locally on every applicable gate: - -- `make test-mac` — native build and tests on macOS. -- `make test-linux` — inside Docker: a **glibc arm64** base image (Ubuntu or Debian, - never Alpine), run with `--shm-size=512m`. If Docker is not running, STOP and tell - the user — never fake or skip the Linux leg. - -There is no deferred-platform table in this build: a gate is not PASS until both -applicable legs pass. - -**Known issue carried over from a prior run:** gcc-13's TSan is incompatible with -Ubuntu 24.04's default ASLR entropy ("unexpected memory mapping" crash at startup). -Inside the container, run TSan binaries under `setarch -R` (per-process ASLR off). -This is a documented workaround, not a report suppression. Do not burn escalation -attempts rediscovering it. - -## Repository layout (expected) - -``` -docs/Shuttle_SRS.md — locked v1.0 requirements baseline -docs/Shuttle_Implementation_Plan.md — the 7-phase plan with verification gates -PROGRESS.md — your ledger (you create and maintain this) -BLOCKED.md — escalation file (only exists if you are stuck) -``` - -Read all docs/ files plus PROGRESS.md before doing anything. If PROGRESS.md does not -exist, this is iteration zero: verify the environment (Xcode CLT, cmake, Docker Desktop -running and able to pull the base image), `git init`, create PROGRESS.md from the -template below, and begin Phase 0. - ---- - -## DESIGN AMENDMENTS — these OVERRIDE the SRS and plan where they conflict - -The SRS and plan were reviewed before this build started. Four corrections are binding. - -### A1. Cursor model: read / write / watermark — NOT shared A/B fields - -Appendix A's shared `a_start/a_size/b_size` violates single-writer ownership (two fields -have two writers). Do not implement it. Instead the shared BipBuffer state is exactly -three absolute offsets, each strictly single-writer: - -- `write` — producer-owned: end of committed data -- `watermark` — producer-owned: end of valid data before a wrap (set when wrapping early) -- `read` — consumer-owned - -Invariant: if `write >= read`, valid data is `[read, write)`; if `write < read`, valid -data is `[read, watermark)` then `[0, write)`. The A→B handoff is the consumer observing -`read == watermark` while `write < read`, then storing `read = 0` — a single owned-variable -update. Regions A/B exist only as derived concepts in Phase 2 logic, never as shared atomics. -Producer's `reserve_*` state is process-local; do not place it in the segment. -Prior art to consult for the invariants: James Munns' bbqueue. - -### A2. ThreadSanitizer CANNOT detect cross-process races — restructure the TSan gates - -TSan shadow state is per-process; two TSan'd processes sharing a segment are invisible to -each other. Therefore: -- The **TSan gate** runs in `--single-process` mode: producer thread + consumer thread in - one process, against a real `MAP_SHARED` mapping, exercising the identical code paths. -- The **two-process gate** verifies what only it can: pshared init, offset/mapping - correctness, single-init publication — via byte-exact FIFO stress (≥1 GB random sizes), - asymmetric-speed stress both directions, and wrap-heavy stress. No TSan claims attach - to the two-process run. -- Additionally: write the happens-before argument for every shared atomic as an inline - comment block (who stores with what ordering, who loads with what ordering, what it - guarantees). Treat FR-17 as satisfied by single-process TSan + the written argument. - -### A3. Heartbeat + timedwait is the PRIMARY liveness mechanism on BOTH platforms - -A robust mutex only fires if the peer died *holding the mutex*. The common death (peer -SIGKILLed on the lock-free path while the survivor is parked in `cond_wait`) means nobody -ever signals — a plain `cond_wait` deadlocks forever, on Linux too. Therefore: -- ALL parked waits on ALL platforms are `pthread_cond_timedwait` + heartbeat-staleness - check on each timeout. This is core, not a macOS shim. -- Linux robust mutex (`PTHREAD_MUTEX_ROBUST` + `EOWNERDEAD` → repair state → - `pthread_mutex_consistent` → unlock, in that order) is *additional* hardening for the - narrow died-holding-the-lock window. -- macOS has no `pthread_mutex_timedlock`: acquiring the park mutex on macOS must be a - `pthread_mutex_trylock` loop with short sleep + heartbeat check, never a bare lock. -- Crash tests need TWO kill points: (1) mid-reservation on the lock-free path (exercises - heartbeat timeout), (2) while holding the park mutex (exercises robust recovery on - Linux / trylock-loop escape on macOS). - -### A4. The parking protocol requires seq_cst, not just recheck-under-mutex - -The flag-then-recheck dance has a Dekker-style store→load hole: consumer's stale -"empty" recheck and producer's stale `waiting==0` load can coexist under acquire/release. -Use `memory_order_seq_cst` for the four accesses in the park decision (waiting-flag -store and load; the cursor publish and the recheck that gate parking/signaling) — or -equivalently `atomic_thread_fence(seq_cst)` after the flag store and after the commit. -Data-path cursors stay release/acquire in the fast case. Comment this as thoroughly as -the wrap handoff. - -### Minor amendments (also binding) - -- `pthread_cond_timedwait` must not use CLOCK_REALTIME: `pthread_condattr_setclock(CLOCK_MONOTONIC)` - on Linux; `pthread_cond_timedwait_relative_np` on macOS. Behind the `platform.hpp` seam. -- Pad each hot atomic (`read`, `write`/`watermark`, waiting flags) to its own **128-byte** - cache line (Apple Silicon line size; also correct on x86). `static_assert` offsets. -- Phase 0 includes gate **G0.4**: a two-process pshared mutex + condvar smoke test - (waiter parks on a condvar in shm; signaler wakes it; timeout = failure; per-run nonce - against stale segments). On THIS host the macOS leg of G0.4 is the single highest-risk - unknown in Phase 0 — macOS pshared-condvar support is historically spotty. If it fails - on macOS, the fallback is `os_sync_wait_on_address` (macOS 14.4+) behind the platform - seam; record the decision in PROGRESS.md. -- Phase 7 adds a **Unix domain socket** baseline alongside HTTP (raw binary over UDS, - same workloads). Latency timestamps cross processes via CLOCK_MONOTONIC written into - the payload header. Numbers measured inside Docker on this host are labeled - "virtualized — not headline figures"; headline NFR-P1 claims require bare-metal Linux - (record as provisional until then). Native macOS numbers are reportable as macOS-dev - figures. -- Python wrapper: the borrowed `memoryview` must be invalidated on `release_read` (wrap in - an object that raises after release); document that Python cannot enforce the borrow. -- The G0.4 smoke test — and every other multi-process test — must launch its processes via - **fork+exec or `posix_spawn`**: the driver runs the test binary twice with role arguments - (e.g. `--role=waiter` / `--role=signaler`). Plain `fork` without `exec` is **forbidden**: - TSan on macOS does not support fork-without-exec and the child inherits a broken runtime. - (Added 2026-06-10; recorded in PROGRESS.md decisions log.) - ---- - -## LOOP PROTOCOL — one iteration, every iteration - -1. **Orient.** Read PROGRESS.md. Identify the lowest-numbered unmet gate. That gate is - this iteration's sole objective. Never work ahead of an unmet gate. -2. **Work** toward that gate. Small commits, one logical change each, message format: - `phase{N}: [G{N}.{M}]`. -3. **Verify.** Run the gate's check on BOTH legs (`make test-mac` and `make test-linux`) - where the gate applies to both. A gate passes only if sanitizer-clean where the plan - says so. -4. **Record.** Update PROGRESS.md: gate status (PASS with command + date / FAIL with the - exact failure), decisions made, next objective. The ledger must let a fresh session - resume with zero other context. -5. **Stop conditions for this iteration:** a gate just passed (commit, record, end the - iteration cleanly), or you hit the escalation rule below. - -### Scheduled-job hygiene - -If this build is driven by a recurring scheduled job, record the job ID and its expiry -date in PROGRESS.md. If the expiry is within 24 hours at the start of an iteration, -re-register the job and update the ledger. If all gates are PASS, cancel the job, write -the final summary in PROGRESS.md, and stop. - -### Escalation rule - -If the same gate fails after **3 distinct, materially different attempts** (not retries -of the same idea), STOP. Write BLOCKED.md containing: the gate, the three hypotheses -tried, the exact failing output, and your current best theory. Do not weaken the test, -do not skip the gate, do not continue to later phases. End the iteration. If BLOCKED.md -exists at the start of an iteration, do no work: report its contents and stop. - -### Hard rules (never violate, even to make progress) - -- **Never modify a test/gate to make it pass.** Gates change only if PROGRESS.md records - a reasoned amendment consistent with the SRS and the amendments above. -- **Never suppress or filter sanitizer reports.** A TSan/ASan finding is a bug until - proven a documented false positive (record the proof; `setarch -R` for the known - TSan/ASLR incompatibility is pre-approved). -- **One variable per phase.** If a gate fails, the bug is in this phase's new variable; - do not refactor already-gated code without recording why. -- **Offsets, never pointers, in the segment.** From line one. -- All platform divergence lives in `platform.hpp`. Nothing else may `#ifdef` on platform. -- Stay inside this repository (plus the Docker container). Do not modify global config - or install host-wide packages without recording it as a ledger decision. - -## PROGRESS.md template (create on iteration zero) - -```markdown -# Shuttle Build Ledger -## Current objective: G0.1 -## Scheduled job: (id, cadence, expiry) -## Gate status -| Gate | mac leg | linux leg | Verified by | Date | Notes | -|------|---------|-----------|-------------|------|-------| -| G0.1 | PENDING | PENDING | | | | -... (all gates G0.1–G7.3, plus G0.4 and the second crash kill-point as G5.4) -## Decisions log -## Session notes (newest first) -``` - -## Phase order and gate inventory - -Phases 0–7 and gates G0.1–G7.3 are exactly as specified in -docs/Shuttle_Implementation_Plan.md, modified by the amendments above: -- G0.4: two-process pshared mutex+condvar smoke passes natively on macOS AND in the - container. The macOS leg gates Phase 4. -- G3.1 (modified): byte-exact ≥1 GB two-process stress; TSan-clean applies to the - single-process dual-thread configuration of the same code. -- G5.x (modified): heartbeat/timedwait gates run on BOTH platforms; G5.4 (new) = - kill-while-holding-mutex recovery on both platforms. -- G7.x (modified): pass criteria measured against BOTH the HTTP baseline and the UDS - baseline; report both ratios, with the virtualization labeling rule above. - -Begin. diff --git a/docs/Shuttle_Implementation_Plan.md b/docs/Shuttle_Implementation_Plan.md deleted file mode 100644 index ce408af..0000000 --- a/docs/Shuttle_Implementation_Plan.md +++ /dev/null @@ -1,210 +0,0 @@ -# Shuttle — Implementation & Sprints Plan - -**Companion to:** Shuttle SRS v1.0 (locked baseline) -**Author orientation:** dev on Apple M3 (macOS), ship on Linux (arm64 + x86_64) -**Guiding principle:** *isolate one variable per phase.* Pure data-structure logic is debugged before concurrency; concurrency before IPC; the lock-free data path before the wake mechanics; the wake mechanics before crash recovery. Never debug two of these at once — that is how shared-memory projects die. - ---- - -## The build-order thesis (read first) - -The single biggest risk in this project is conflating bugs. A wrong answer can come from (a) BipBuffer arithmetic, (b) a memory-ordering mistake, (c) a lost wakeup, (d) a pshared-init mistake, or (e) a platform difference. If you bring up everything at once and the consumer reads garbage, you cannot tell which of the five it is. - -So the plan deliberately stages capability so that **at every phase, only one of those five can be the culprit:** - -| Phase | New variable introduced | Everything before it is already trusted | -|---|---|---| -| 0 | toolchain, sanitizers, containers | — | -| 1 | segment lifecycle + header | toolchain | -| 2 | BipBuffer arithmetic (single thread, no shm) | nothing concurrent yet | -| 3 | atomics + cross-process ordering (busy-poll, **no mutex**) | BipBuffer logic is proven | -| 4 | parking-lot wake (mutex + condvars) | ordering is proven | -| 5 | crash recovery (heartbeat → robust) | wake is proven | -| 6 | C ABI freeze + FFI | core is proven | -| 7 | benchmark | everything is proven | - -A corollary: **Phase 3 is the hard one.** Budget accordingly. If something is going to eat a week, it is cross-process release/acquire ordering on the A→B handoff, debugged under a multi-process ThreadSanitizer. - ---- - -## Phase 0 — Toolchain, sanitizers, and the two-platform harness - -**Goal:** a CMake project that compiles a stub library and runs an empty test on (1) macOS arm64 natively and (2) Linux arm64 inside Docker — *before any real code exists.* Establishing both targets now means every later phase is validated on both with one command. - -### Work items (in order) -1. CMake skeleton, C++17, three targets: `shuttle_core` (static lib), `shuttle_tests`, `shuttle_bench`. Keep the core free of any test/bench dependency. -2. Create `platform.hpp` — the single seam where `#ifdef __linux__` / `#ifdef __APPLE__` lives. Nothing else in the codebase is allowed to `#ifdef` on platform. This file is the entire macOS-vs-Linux strategy in one place. -3. Sanitizer presets wired from day one: an ASan preset and a **separate** TSan preset (ASan and TSan are mutually exclusive — you cannot link both). Add a UBSan flag to the ASan build. -4. **Docker for the Linux target.** Docker Desktop on Apple Silicon runs `linux/arm64` containers natively in its lightweight VM — no QEMU emulation — so robust-mutex work in Phase 5 runs at near-native speed on your M3. Two concrete gotchas to bake into the Dockerfile/run now: - - **Use a glibc base (Ubuntu/Debian arm64), not Alpine/musl.** musl's `PTHREAD_MUTEX_ROBUST` support is historically weaker and divergent; you do not want to discover that in Phase 5. glibc gives you the canonical `EOWNERDEAD` semantics the SRS assumes. - - **`--shm-size` defaults to 64 MB inside containers.** With 50 MB payloads and the SRS's ~2× capacity rule, your segment is >100 MB and `ftruncate`/`mmap` will fail with a confusing `ENOSPC`/`EINVAL`. Set `docker run --shm-size=512m` (or mount a tmpfs) now and document it. -5. A trivial smoke target that maps and unmaps 4 KB of shm on both platforms, proving the toolchain end-to-end. - -### Debugging strategy -- Make `make test-mac` and `make test-linux` (the latter wraps `docker run`) single commands. If running both targets is more than one keystroke, you will stop doing it, and the platform gap will bite you late. - -### Verification gates -- **G0.1** Empty test binary builds and runs clean under ASan on macOS and under ASan in the arm64 container. -- **G0.2** TSan preset links and runs the empty test on both (proves TSan instrumentation is wired before you need it in Phase 3). -- **G0.3** The 4 KB smoke target maps/unmaps successfully in the container with a >100 MB `--shm-size`, proving the shm-size gotcha is handled. - ---- - -## Phase 1 — Segment & header architecture - -**Goal:** create, open, validate, close, and unlink a named segment with a correct, version-checked, properly-aligned header. No BipBuffer, no concurrency — just the memory object and its metadata. - -### Work items (in order) -1. **Define the header struct** (App. A) using only fixed-width types (`uint64_t`, `uint32_t`, `std::atomic`, etc.). Add explicit padding so `data_offset` lands on a cache-line boundary (64 B). `static_assert` the struct size and key field offsets so a layout change can't silently break the ABI or the other-language readers. -2. **Offsets, never pointers (CON-3 / FR-16).** Establish the discipline *now*, before there is anything to point at: a tiny `resolve(base, offset)` helper and a rule that nothing of pointer type is ever stored in the segment. This is cheap to honor from line one and expensive to retrofit. -3. **Lifecycle:** `shuttle_create` → `shm_open(O_CREAT|O_EXCL)` + `ftruncate` + `mmap(MAP_SHARED)` + header init. `shuttle_open` → `shm_open` (no create) + `mmap`, *no* re-init. `shuttle_close` → `munmap` + free local handle. `shuttle_unlink` → `shm_unlink`. -4. **Magic + version validation (FR-3)** on open; distinct error on mismatch. -5. **Capacity validation (FR-4):** reject `capacity < max_payload + framing` with a distinct error. -6. **The single-init problem (App. B #5):** the creator initializes the header; an opener must not read BipBuffer/primitive state until init is *published*. Use an `atomic init_state` written last with a release store; the opener spins/acquire-loads until it sees `READY`. (A full seqlock is overkill for one-shot init.) -7. **Platform-specific seam:** macOS `shm_open` names are capped (~31 chars incl. leading `/`) and an shm object can effectively be `ftruncate`d only once at creation. Enforce a short-name check and one-shot truncate in `platform.hpp`. - -### Debugging strategy -- On Linux, `ls -l /dev/shm` is your ground truth — every create should appear there, every unlink should remove it. On macOS there is no `/dev/shm`; rely on create/open succeeding from a second process and on clean teardown. -- Write a 5-line `shuttle_inspect` CLI that opens a segment and dumps the header. You will use it constantly through Phase 5. - -### Verification gates -- **G1.1** Process A creates; process B opens, reads matching magic/version; B with a bumped version gets the mismatch error. -- **G1.2** `create` with `capacity < max_payload + 8` fails with the FR-4 error code, not a crash. -- **G1.3** Leak check (NFR-R2): after create→close→unlink, `/dev/shm` is clean on Linux; the name cannot be re-opened on either platform. - ---- - -## Phase 2 — BipBuffer logic (single-threaded, no shared memory) - -**Goal:** a provably-correct BipBuffer (D2) and 8-byte length framing (§2.4) running against a plain heap buffer in **one thread**. Zero concurrency, zero IPC. This is the phase where you nail the arithmetic that everything else assumes is correct. - -### Work items (in order) -1. Implement `reserve(n) / commit(n) / read_block() / release(n)` over a `std::byte*` and plain `size_t` cursors (not atomics yet — concurrency is Phase 3). Model region A, region B, and the producer-private reserve cursor exactly as in App. A. -2. The **early-wrap rule (App. B #6):** reserve `8 + len` as one unit; if it doesn't fit after A, wrap to B at offset 0; never split a payload. Region A therefore always ends on a message boundary. -3. The **A→B handoff:** when A fully drains, `A := B`, clear B. Get this dead right here, with no memory-ordering noise to confuse you, because in Phase 3 this exact transition becomes the most delicate ordering point in the whole system. -4. Length framing: write `u64 len` then payload; reader peeks the length and returns `(ptr, len)` into the contiguous payload (FR-7). - -### Debugging strategy -- This phase is pure logic, so it is the **one place you can use heavy fuzzing/property testing.** Drive thousands of random-sized writes interleaved with random-sized drains; after each operation assert the invariants: A and B never overlap, no reservation ever crosses the physical end, total bytes in == total bytes out, FIFO order preserved. A property test here is worth more than any amount of staring at Phase 3. - -### Verification gates -- **G2.1** Byte-exact FIFO over ≥100k random write/read pairs of random sizes; invariants hold after every op. -- **G2.2** The three named edge cases: payload that *exactly* fills the space after A; a payload that forces an early wrap to B; a max-size payload (`= max_payload`). -- **G2.3** A write of `max_payload + 1` (logically, > usable capacity) fails fast rather than looping or wrapping incorrectly. - ---- - -## Phase 3 — Lock-free hot path, cross-process, **busy-poll** (the hard phase) - -**Goal:** move the BipBuffer state into the shared segment as atomics and make two *separate processes* exchange data correctly with the lock-free cursor protocol — **with no mutex and no condvars yet.** When the buffer is empty/full, the waiting side simply spins (busy-polls). This deliberately separates *memory-ordering correctness* from *wake mechanics*; you debug them one at a time. - -### Work items (in order) -1. Promote the cursors to `std::atomic` in the header. Enforce the **single-writer ownership** model from App. A: producer is the only writer of `a_size`, `b_size`, `reserve_*`; consumer is the only writer of `a_start`. Single-writer-per-atomic is what makes the lock-free path sound. -2. Apply the ordering contract: producer publishes a commit with a **release** store on the size it grew; consumer observes it with an **acquire** load before reading payload bytes. Symmetric for the consumer's `a_start` release and the producer's acquire. The release/acquire pair is what guarantees the payload bytes are visible before the cursor that advertises them. -3. **The A→B handoff is the ordering hotspot.** The consumer setting `a_start = 0`, adopting `b_size` as the new `a_size`, and clearing `b_size` must be ordered so the producer never sees a torn intermediate state. Specify the exact fence sequence in the design doc and comment it inline; this is the line of code most likely to harbor a 1-in-10⁶ bug. -4. Empty/full handling: **spin** (optionally with a `pause`/`yield` hint). No blocking primitives yet. This is intentionally CPU-wasteful and temporary. - -### Debugging strategy -- **Multi-process ThreadSanitizer is the whole game this phase.** TSan across processes requires the shared segment be mapped in both and both binaries built with TSan; run producer and consumer as separate TSan'd processes against the same segment. A clean TSan run over a long stress test is your proof of the ordering contract — far stronger than "it didn't crash." -- Asymmetric-speed stress: run producer fast / consumer slow (forces full → backpressure-by-spin), then the reverse (forces empty → spin), then both flat-out (maximizes handoff contention). The handoff bug, if present, surfaces under flat-out. -- Keep a `--single-process` mode (producer thread + consumer thread, same address space) as a faster inner-loop check, but **the gate is the two-process run** — same-process testing hides pshared/mapping bugs. - -### Verification gates -- **G3.1** Two separate processes exchange ≥1 GB of data in random-sized messages, byte-exact, FIFO — **clean under multi-process TSan.** -- **G3.2** Producer-fast and consumer-fast asymmetric stress both pass byte-exact (proves spin-based backpressure and spin-based empty handling are correct). -- **G3.3** A targeted A→B-handoff stress (capacity sized so wraps happen every few messages) passes byte-exact under TSan. - ---- - -## Phase 4 — Parking-lot wake (mutex + condvars, off the hot path) - -**Goal:** replace the busy-poll from Phase 3 with real blocking, so an idle peer consumes ~0% CPU — while keeping the mutex strictly off the normal path (§2.3). Because Phase 3 already proved ordering, any bug here is *by construction* a wake bug (lost wakeup, missed signal), which is a much smaller search space. - -### Work items (in order) -1. Add the pshared `pthread_mutex_t lock` and the two `pthread_cond_t` (`not_empty`, `not_full`) to the segment. **Initialize the pshared attribute explicitly** on all three (App. B #2) — a default-initialized mutex silently "works" within one process and fails across processes, which is a nasty false-positive in same-process tests. -2. Add the `consumer_waiting` / `producer_waiting` atomic flags. -3. Block path (consumer empty): set `consumer_waiting`; lock; **re-check emptiness under the lock** (lost-wakeup guard, App. B #7); `pthread_cond_wait(not_empty)`; on wake clear the flag, unlock, retry the Phase-3 lock-free read. Symmetric for producer-full on `not_full`. -4. Wake path (kept off hot path): after a commit, the producer does an atomic load of `consumer_waiting`; only if set does it briefly lock + `signal(not_empty)`. In the common case (peer not sleeping) neither side ever touches the mutex. Symmetric on release. - -### Debugging strategy -- The failure mode here is the **lost wakeup**: peer checks "empty", then the other side commits + signals *before* the peer is waiting, peer then waits forever. The flag-then-recheck-under-lock dance prevents it. Stress it by inserting artificial scheduling delays (a `sleep(0)`/jitter) between the predicate check and the wait, which widens the race window and makes a missing guard reproduce quickly. -- CPU usage is now a *correctness* signal: an idle blocked peer must sit near 0% CPU. If it's spinning, your block path isn't engaging. - -### Verification gates -- **G4.1** Idle consumer (no data for several seconds) shows ~0% CPU — proves true blocking, not residual spin. -- **G4.2** "Trickle" stress: producer sends one message every random interval; consumer receives every message with no lost/extra wakeups over ≥100k messages. -- **G4.3** Wake latency: p99 from commit→consumer-has-payload is microsecond-scale on target hardware (NFR-P3); the hot path still touches no mutex when the peer isn't parked (verify by counting lock acquisitions). - ---- - -## Phase 5 — Crash resilience: macOS heartbeat first, then Linux robust - -**Goal:** a peer that dies mid-transfer never permanently strands the survivor. Per D4 this is a *hard* guarantee on Linux and a *best-effort* guarantee on macOS. **Build the macOS path first** because it's your dev machine and lets you exercise the abstraction immediately; then move into the arm64 Linux container for the robust path. - -### Step 5a — macOS heartbeat (on the M3, dev-only guarantee — FR-19) -1. Add `producer_heartbeat` / `consumer_heartbeat` monotonic atomics to the header; each side bumps its counter on activity. -2. macOS lacks `pthread_mutex_timedlock` and robust mutexes. So all blocking waits become `pthread_cond_timedwait`; on each timeout, check whether the peer's heartbeat has gone stale past a configurable threshold. Stale ⇒ declare the peer dead, abort the wait with a distinct error rather than blocking forever. -3. Put **all** of this behind the `platform.hpp` seam — a `wait_blocking()` and a `recover_if_peer_dead()` interface with two implementations. The core code above this seam must not know which platform it's on. - -### Step 5b — Linux robust mutex (in the arm64 glibc container — FR-18) -1. Initialize the segment mutex with `PTHREAD_MUTEX_ROBUST` (in addition to pshared). -2. On `lock()` returning `EOWNERDEAD`: **repair the small, well-defined protected state, *then* call `pthread_mutex_consistent`, *then* unlock** — in that order (App. B #3). Get it backwards and the mutex is permanently dead. Because the critical section guards only park/wake bookkeeping (not a data copy), "repair" is genuinely small. -3. Combine with single-writer cursor ownership: the survivor can also conclude the peer is gone and tear the channel down cleanly. - -### Debugging strategy -- The crash test is `SIGKILL` (not `SIGTERM` — you want no cleanup) on the producer while a reservation is in flight. Script it so it's repeatable: producer reserves, raises a flag, sleeps; harness kills it at that exact point. -- On Linux, deliberately write a *buggy* recovery once (call `consistent` before repairing) to confirm your test actually detects a permanently-dead mutex — i.e., verify the test can fail. - -### Verification gates -- **G5.1 (macOS)** `SIGKILL` the producer mid-reservation; the consumer's blocked wait aborts via heartbeat staleness with the documented error within ~the configured threshold (FR-19). -- **G5.2 (Linux container)** Same kill; the survivor's `lock` returns `EOWNERDEAD`, recovers via `pthread_mutex_consistent`, and the channel never permanently deadlocks (FR-18 / NFR-R1). -- **G5.3** No shm object leaks after a crashed run on either platform once the survivor tears down (NFR-R2). - ---- - -## Phase 6 — C ABI freeze + Rust/Python FFI - -**Goal:** lock the `extern "C"` surface and prove the same segment is driven byte-exact from C++, Rust, and Python. The core is already trusted, so any failure here is a *binding/marshalling* bug, not a core bug. - -### Work items (in order) -1. Freeze the signatures from SRS §3.1, the integer error-code enum, and the flags. **No C++ exception may cross the boundary (IF-1):** wrap every entry point in a `try/catch` that converts to an error code. -2. Ship a single versioned C header (IF-4) with the struct/enum/error definitions — this is the source of truth for both `bindgen` and `cffi`. -3. Rust: generate FFI with `bindgen`, then a thin safe wrapper. The borrow path (`acquire_read` → pointer → `release_read`) is where Rust lifetimes earn their keep — model the borrowed slice's lifetime so it cannot outlive `release_read`. -4. Python: `cffi` (preferred over `ctypes` for header-driven binding). Expose the borrowed payload as a zero-copy `memoryview` over the returned pointer so you don't accidentally copy on the Python side and silently lose the headline benefit. - -### Debugging strategy -- Test cross-language pairs against the *same running segment*, not language-internal mocks: a C++ producer process with a Python consumer process, and a C++ producer with a Rust consumer (FR-21). A round trip that's byte-exact across a language boundary is the real proof. -- Watch for accidental copies in the dynamic languages — a `bytes(...)` in Python or a `.to_vec()` in Rust on the borrow path defeats the zero-copy contract. Assert pointer identity / use a profiler to confirm no copy. - -### Verification gates -- **G6.1** C++ producer ↔ Python consumer: byte-exact round trip over the borrow path (FR-21). -- **G6.2** C++ producer ↔ Rust consumer: byte-exact, and the Rust wrapper *fails to compile* if borrowed data is used after `release_read` (lifetime correctness). -- **G6.3** An induced error (e.g., open a nonexistent segment) surfaces as the correct integer code in all three languages, with no exception/panic escaping the ABI. - ---- - -## Phase 7 — The headline benchmark - -**Goal:** prove NFR-P1/P2/P3 against a *fair* baseline (D7): raw uncompressed binary payload as the HTTP body, keep-alive on, sensible socket buffers — HTTP doing the least wasteful thing it can. - -### Work items (in order) -1. Two transports, identical workload: (a) Shuttle borrow path; (b) raw-binary localhost HTTP. -2. Two workloads: a single **50 MB** blob (tensor/context case) and a **16 KB** frame stream at a fixed rate (audio case, D5 secondary). -3. Methodology: warm-up iterations discarded; then many iterations; report **median and p99**. Measure end-to-end latency (producer commit → consumer holds the full payload), per-side CPU attributable to copy/serialize (profiler), and streaming throughput (MB/s, frames/s). -4. **Where to run it:** correctness across Phases 3–6 is fully valid in the arm64 container, but the *performance numbers* should be taken on real Linux hardware where possible — a virtualized container kernel adds overhead that can muddy the ratio. Report macOS-dev and Linux-prod figures separately and label them (D4 / §4 controls). - -### Verification gates (these are the SRS pass criteria) -- **G7.1** 50 MB end-to-end latency ≥ **10×** lower than the HTTP baseline (NFR-P1; stretch 50×). -- **G7.2** Profiler shows negligible copy/serialize CPU on the borrow path (NFR-P2) — the consumer is genuinely reading the producer's bytes in place. -- **G7.3** Microsecond-scale wake latency under load (NFR-P3), consistent with the G4.3 measurement. - ---- - -## Cross-cutting reminders (pin these above your desk) - -- **One variable per phase.** If a phase's gate fails, the bug is almost certainly in *that* phase's new variable — don't go spelunking in already-gated code. -- **Offsets, never pointers**, from Phase 1 line one. -- **Two-process TSan run is the real gate**, not the same-process convenience test — same-process testing hides pshared and mapping bugs. -- **The A→B handoff (Phase 3) and the EOWNERDEAD recovery order (Phase 5b)** are the two places a subtle, rare bug will hide. Over-comment both. -- **Glibc container + `--shm-size`** are settled in Phase 0 so they never surprise you later. -- Re-run **every** phase's gates on **both** platforms before declaring it done — the whole point of the Phase 0 two-command harness. diff --git a/docs/Shuttle_SRS.md b/docs/Shuttle_SRS.md deleted file mode 100644 index 63f2805..0000000 --- a/docs/Shuttle_SRS.md +++ /dev/null @@ -1,331 +0,0 @@ -# Software Requirements Specification - -## Shuttle — A High-Performance, Zero-Copy Shared-Memory IPC Framework for Local AI Workloads - -**Version:** 1.0 — Final design baseline -**Status:** Locked for implementation -**Orientation:** Production-grade portfolio piece. Pragmatics, optimized C++17, and a bulletproof benchmark are weighted above formal standards conformance; requirement IDs are retained for traceability. - -> *Shuttle* — in a loom, the shuttle carries the thread back and forth across the warp. Here it carries data buffers back and forth between processes. (Fits the Loom / Whetstone family; rename freely.) - ---- - -## Locked Architectural Decisions (v1.0) - -These seven decisions are settled and drive every requirement below. - -| # | Decision | Consequence | -|---|---|---| -| D1 | **Topology:** strictly SPSC, point-to-point, one-way per channel. Multi-process stacks instantiate multiple pairwise channels (Rust→Python, Python→C++, …). | Each channel has exactly one writer and one reader, enabling a lock-free data path. | -| D2 | **Buffer:** a **Bipartite Buffer (BipBuffer)**, not a plain circular ring. | Every reserved/borrowed block is guaranteed *physically contiguous* — no mid-payload wrap — so the zero-copy pointer handoff is always valid. | -| D3 | **Backpressure:** block the producer. Data integrity is non-negotiable for embeddings/LLM context. | Producer sleeps until enough *contiguous* space exists; nothing is ever dropped. | -| D4 | **Platforms:** develop on macOS (Apple silicon), ship on Linux. Full `PTHREAD_MUTEX_ROBUST` on Linux; documented timeout + heartbeat on macOS. | Crash recovery is a hard guarantee on Linux, a best-effort guarantee on macOS. | -| D5 | **Workload:** large variable-sized blobs (≈50 MB tensors / context windows) dominate; audio-frame streaming is secondary. | Capacity sizing and contiguous reservation are first-class concerns. | -| D6 | **Payload:** opaque bytes. The application agrees on layout (dtype, shape, sample rate) out-of-band. | Shuttle carries transport-level length framing only; it never interprets payload content. | -| D7 | **Benchmark baseline:** raw, uncompressed binary payloads over localhost HTTP. | The comparison is fair and resistant to "straw-man" objections. | - ---- - -## 1. Introduction - -### 1.1 Purpose - -This document specifies **Shuttle**, a C++17 inter-process communication library that moves large binary payloads between cooperating processes on a single machine with near-zero copy overhead, for local AI workloads — audio streaming, vector embeddings, and LLM context windows — across a polyglot stack (a C++ execution engine, a Rust/Tauri frontend, Python sidecars). - -### 1.2 Scope - -Apps such as *Loom* and *Whetstone* currently move data over loopback networking (localhost HTTP). On that path a payload is copied into a kernel socket buffer, traverses the loopback stack, is copied into the receiver's socket buffer, and is framed/deframed by HTTP — several full copies plus protocol overhead per message. For a 50 MB payload this wastes both latency and CPU. - -Shuttle replaces that path for **same-host** communication: one region of physical RAM is mapped into every participating process via POSIX shared memory, and access is coordinated with synchronization primitives stored *inside* that region. A producer writes a payload once; a consumer reads it in place. No copy crosses the network stack, and no per-message serialization is needed for payloads already in a flat binary layout (PCM audio, `float32` tensors, byte blobs). - -**In scope (v1.0):** a single-producer/single-consumer, one-way streaming channel (D1) built on a BipBuffer (D2); channel lifecycle (create/open/close/unlink); a lock-free data path with a parking-lot blocking mechanism; producer backpressure (D3); a stable C ABI callable from C++, Rust (FFI), and Python (`cffi`/`ctypes`); a zero-copy borrow path plus a copy-out convenience path; per-platform crash resilience (D4); and a reproducible benchmark harness versus a raw-binary localhost-HTTP baseline (D7). - -**Out of scope (v1.0):** cross-machine transport; Windows (different API — see App. C); multi-producer/multi-consumer and pub/sub (App. D); encryption of the segment; automatic serialization of structured objects (D6). - -### 1.3 Definitions - -| Term | Definition | -|---|---| -| **SPSC** | Single-Producer, Single-Consumer: exactly one writer and one reader per channel (D1). | -| **BipBuffer** | Bipartite buffer: a circular-buffer variant that serves allocations from at most two contiguous regions so that any single reserved block never wraps the physical end of the buffer. | -| **Zero-copy borrow** | The consumer (or producer) receives a pointer directly into the segment and reads/writes in place; the payload is never duplicated. | -| **Lock-free hot path** | Normal-case reads/writes coordinate via atomic cursors with acquire/release ordering and take no mutex. | -| **Parking-lot wake** | The mutex + condition variables are used *only* to sleep a process that must block and to wake it; they are off the hot path. | -| **Robust mutex** | `PTHREAD_MUTEX_ROBUST`: if the owner dies holding it, the next acquirer gets `EOWNERDEAD` and can repair state via `pthread_mutex_consistent` (Linux). | -| **Heartbeat** | A monotonic liveness counter each side updates, used on macOS to detect a dead peer where robust mutexes are unavailable (D4). | -| **Transport framing** | Shuttle's internal per-message length prefix, distinct from application payload semantics (D6). | -| **Backpressure** | Flow control that blocks the producer when the buffer cannot accept a write (D3). | -| **`shm_open` / `mmap`** | POSIX calls to create/open a named shared-memory object and map it into a process. | - -### 1.4 References - -POSIX.1-2017 (`shm_open`, `mmap`, `pthread_*`); Linux man-pages `shm_overview(7)`, `pthread_mutexattr_setpshared(3)`, `pthread_mutexattr_setrobust(3)`; Simon Cooke, "The Bip Buffer" (origin of the bipartite-buffer technique); ISO/IEC/IEEE 29148:2018 (structure reference only). - ---- - -## 2. Architecture - -This section is the heart of the document; §3 formalizes it into testable requirements. - -### 2.1 Topology (D1) - -A channel is a one-way pipe with one writer and one reader. The three-process stack is composed of several such channels: - -``` - Rust / Tauri ──[chan: ui_to_py]──► Python sidecar ──[chan: py_to_cpp]──► C++ engine - ▲ │ - └───────────────────────[chan: cpp_to_ui]───────────────────────────────┘ -``` - -Each channel is independent: its own named segment, its own BipBuffer, its own primitives. Reverse or request/response flows are simply two channels. There is no shared global state across channels. - -### 2.2 Segment and the BipBuffer (D2, D5) - -A segment is one `mmap`'d region: a fixed **control header** followed by the **data region**. The data region is managed as a BipBuffer rather than a plain ring, because a plain ring lets a single logical payload straddle the wrap point — which would force either a two-part read or an internal copy to reassemble, breaking the zero-copy contract. The BipBuffer guarantees that every reserved write block and every readable block is one contiguous run of bytes. - -**BipBuffer mechanics (single writer, single reader):** the buffer tracks a primary readable region **A** and, after a wrap, a secondary region **B** anchored at offset 0. - -- **Reserve(n)** (producer): if space after A to the physical end is sufficient, reserve there (A will grow). Otherwise reserve at offset 0, beginning region B (the writer "wraps early" rather than splitting the block). Either way the reserved block is contiguous, or the reservation fails/blocks (D3). -- **Commit(n)** (producer): grows A's size (or B's size) and publishes it. -- **Read block** (consumer): region A, returned as a single contiguous `(ptr, len)`. -- **Release(n)** (consumer): advances A's start. When A is fully drained, **A := B** and B is cleared; subsequent reads continue from what was B. - -Because the writer only ever wraps to B on a whole-message boundary (it reserves `8 + payload` as one unit — see §2.4), region A always ends on a message boundary, so message order and boundaries are preserved naturally. - -**Sizing rule (D3 + D5):** with block-the-producer backpressure, a write that can never be satisfied would block forever. Therefore `shuttle_create` validates `capacity ≥ max_payload + framing`, and a write larger than the usable capacity fails fast with a distinct error rather than blocking. A capacity of roughly **2× the largest payload** is recommended so the producer can fill region B while the consumer drains region A (pipelining) instead of strict ping-pong. - -### 2.3 Synchronization: lock-free hot path + parking-lot wake (D1, D4) - -Strict SPSC means the producer is the sole writer of the write/reserve cursors and the consumer the sole writer of the read cursor. Each cursor therefore has a single writer and can be published with **release** stores and observed with **acquire** loads — no mutex is required on the normal path. - -The mutex and the two condition variables (`not_empty`, `not_full`) exist solely to **park** a process that must block and to **wake** it: - -- **Consumer, buffer empty:** set an atomic `consumer_waiting` flag; take the mutex; re-check emptiness (guard against lost wakeups); `pthread_cond_timedwait(not_empty)`; on wake, clear the flag and retry the lock-free read. -- **Producer, insufficient contiguous space:** symmetric, using `consumer`'s progress and `not_full`. -- **Wake path (kept off the hot path):** after a commit, the producer checks `consumer_waiting` atomically; only if set does it briefly take the mutex and signal `not_empty`. The consumer signals `not_full` after a release symmetrically. In the common case (peer not sleeping) neither side touches the mutex. - -The critical section guarded by the mutex is thus tiny and bounded, which is what makes the crash story (§2.5) cheap. - -> **Pragmatic fallback:** if verifying the fully lock-free cross-process ordering proves too costly for v1.0, a mutex-guarded data path is an acceptable degradation — it changes only the performance profile of small-message streaming, not correctness or the headline 50 MB benchmark. The lock-free path is the target. - -### 2.4 Transport framing vs. payload semantics (D6) - -Shuttle preserves message boundaries by prepending a fixed **8-byte little-endian length** to each reservation: a write of an `L`-byte payload reserves `8 + L` contiguous bytes, writes `L`, then the payload. The reader peeks the 8-byte length (a trivial read, never the payload) and hands back a contiguous pointer to the payload region with length `L`. - -This length is *transport* metadata owned by Shuttle. The payload's meaning — dtype, tensor shape, sample rate, framing of sub-records — is **out-of-band** application convention (D6). Shuttle never reads or interprets payload bytes. - -### 2.5 Crash resilience, per platform (D4) - -Because the only lock is the brief park/wake critical section (§2.3), a peer dying mid-transfer cannot strand a lock held over a data copy. - -- **Linux (production):** the park/wake mutex is `PTHREAD_MUTEX_ROBUST`. If a peer dies holding it, the survivor's `lock` returns `EOWNERDEAD`; the survivor restores the (small, well-defined) protected state and calls `pthread_mutex_consistent`. Combined with single-writer cursor ownership, the survivor can also detect that the peer is gone and tear the channel down cleanly. -- **macOS (development only):** robust mutexes and `pthread_mutex_timedlock` are **not** available. Instead, each side updates a monotonic **heartbeat** counter in the header; blocking waits use `pthread_cond_timedwait` and, on each timeout, check whether the peer's heartbeat has gone stale beyond a configurable threshold. A stale peer is declared dead and the wait aborts with an error rather than blocking indefinitely. This is explicitly a weaker, best-effort guarantee, documented as such. - -### 2.6 Cross-language boundary - -The core is C++17 exposing an `extern "C"` ABI (§3.1). Rust binds via FFI (`bindgen`); Python via `cffi`/`ctypes`. No C++ exceptions cross the boundary; all errors are integer codes. The header uses only fixed-width types and documented alignment so the same bytes are read identically by all three languages. - ---- - -## 3. Specific Requirements - -IDs are stable; priority is **M**ust / **S**hould / **C**ould; each item is individually testable. - -### 3.1 Interface (C ABI) - -Representative sketch (final signatures fixed during implementation): - -```c -typedef struct shuttle_channel shuttle_channel; - -/* lifecycle */ -shuttle_channel* shuttle_create(const char* name, size_t capacity_bytes, - size_t max_payload_bytes, int* err); -shuttle_channel* shuttle_open (const char* name, int* err); -void shuttle_close (shuttle_channel* ch); /* munmap + free local handle */ -int shuttle_unlink(const char* name); /* shm_unlink the named object */ - -/* copy convenience path */ -int shuttle_write(shuttle_channel* ch, const void* data, size_t len, int flags); -long shuttle_read (shuttle_channel* ch, void* out, size_t cap, int flags); - -/* zero-copy borrow path (headline) */ -int shuttle_acquire_write(shuttle_channel* ch, void** ptr, size_t len, int flags); -int shuttle_commit_write (shuttle_channel* ch, size_t actual_len); -int shuttle_acquire_read (shuttle_channel* ch, const void** ptr, size_t* len, int flags); -int shuttle_release_read (shuttle_channel* ch); -``` - -| ID | Pri | Requirement | -|---|---|---| -| IF-1 | M | All public functions use `extern "C"` linkage and C-compatible types; no exception crosses the boundary; errors are returned as codes. | -| IF-2 | M | Both a copy path (`write`/`read`) and a zero-copy borrow path (`acquire`/`commit`/`release`) are provided. | -| IF-3 | M | Blocking vs. non-blocking ("would block") is selectable per call via `flags`. | -| IF-4 | S | A versioned header with stable struct/enum/error definitions is provided for `bindgen` / `cffi`. | - -### 3.2 Functional Requirements - -**Lifecycle** - -| ID | Pri | Requirement | -|---|---|---| -| FR-1 | M | `shuttle_create` creates a named shared object (`shm_open` `O_CREAT`), sizes it (`ftruncate`), maps it (`mmap`, `MAP_SHARED`), and initializes the header, BipBuffer state, and process-shared primitives. | -| FR-2 | M | `shuttle_open` attaches to an existing object and maps it without re-initializing control structures. | -| FR-3 | M | On open, the library validates a magic number and version and returns an error on mismatch. | -| FR-4 | M | `shuttle_create` validates `capacity_bytes ≥ max_payload_bytes + framing`; otherwise it fails with a distinct error (prevents an unsatisfiable, permanently-blocking write — §2.2). | -| FR-5 | M | `shuttle_close` unmaps and releases per-process resources without destroying the named object; `shuttle_unlink` removes the named object so it does not leak across runs. | - -**Topology (D1)** - -| ID | Pri | Requirement | -|---|---|---| -| FR-6 | M | A channel supports exactly one producer and one consumer, one-way. Behavior with a second producer or consumer is undefined and need not be defended in v1.0. | - -**Data transfer over the BipBuffer (D2, D6)** - -| ID | Pri | Requirement | -|---|---|---| -| FR-7 | M | A producer can write a single payload of any length up to `max_payload_bytes`; message boundaries are preserved on the consumer side via internal 8-byte length framing (§2.4). | -| FR-8 | M | Payloads are delivered in FIFO order. | -| FR-9 | M | The zero-copy read path returns a pointer (resolved to the local mapping) into a **contiguous** region for the whole payload, valid until `release_read`; no payload copy occurs. | -| FR-10 | M | The zero-copy write path returns a writable **contiguous** pointer for the reservation; `commit_write` publishes the payload (allowing `actual_len ≤` reserved length). | -| FR-11 | M | The library never interprets payload bytes; only the internal length prefix is read/written by Shuttle (D6). | - -**Backpressure (D3)** - -| ID | Pri | Requirement | -|---|---|---| -| FR-12 | M | When no contiguous region large enough for the reservation exists, a blocking write parks the producer until the consumer frees sufficient contiguous space; a non-blocking write returns "would block". Nothing is dropped. | -| FR-13 | M | When no payload is available, a blocking read parks the consumer on `not_empty` (no busy-poll) until signaled; a non-blocking read returns "would block". | - -**Synchronization (D1)** - -| ID | Pri | Requirement | -|---|---|---| -| FR-14 | M | Primitives (mutex, `not_empty`, `not_full`) live inside the segment, initialized `PTHREAD_PROCESS_SHARED`. | -| FR-15 | M | The normal-case data path is lock-free: cursors are atomics published with release and observed with acquire; the mutex is taken only on the park/wake slow path (§2.3). (Mutex-guarded fallback per §2.3 note is acceptable.) | -| FR-16 | M | All in-segment references are byte **offsets** from the segment base, never absolute pointers (different processes map at different addresses — App. B). | -| FR-17 | M | The control structures are free of data races, verifiable under a multi-process ThreadSanitizer harness. | - -**Crash resilience (D4)** - -| ID | Pri | Requirement | -|---|---|---| -| FR-18 | M | **Linux:** the park/wake mutex is `PTHREAD_MUTEX_ROBUST`; on `EOWNERDEAD` the survivor restores consistent state and calls `pthread_mutex_consistent`; no permanent deadlock results from a peer crash. | -| FR-19 | M | **macOS:** with no robust mutex and no `pthread_mutex_timedlock`, blocking waits use `pthread_cond_timedwait` plus a heartbeat-staleness check; a stale peer aborts the wait with an error. Reduced guarantee is documented. | -| FR-20 | S | The channel exposes liveness/occupancy state (peer-attached, bytes pending, contiguous space free) for application-level dead-peer detection. | - -**Cross-language (D6)** - -| ID | Pri | Requirement | -|---|---|---| -| FR-21 | M | The same channel works end-to-end across languages: a C++ producer with a Python consumer, and a C++ producer with a Rust consumer, are each demonstrated. | -| FR-22 | S | Example Python (`cffi`) and Rust (FFI) wrappers are provided. | - -### 3.3 Non-Functional Requirements - -**Performance** - -| ID | Pri | Requirement | -|---|---|---| -| NFR-P1 | M | For a 50 MB payload, end-to-end latency over Shuttle is ≥ **10×** lower than the raw-binary localhost-HTTP baseline (§4); stretch **50×**. | -| NFR-P2 | M | On the zero-copy path, CPU time spent copying/serializing the payload is effectively zero (consumer reads producer bytes in place), demonstrated by profiling. | -| NFR-P3 | M | Consumer wake latency after a commit is on the order of microseconds, not milliseconds, on target hardware. | -| NFR-P4 | S | Sustained streaming of small audio frames (e.g., 4–64 KB) exceeds the HTTP baseline throughput at a stated frame rate. | - -> **Scope of the claim (state this in the writeup):** Shuttle eliminates kernel-stack copies and HTTP framing. "Zero serialization" holds for payloads already in flat binary layout (PCM, `float32` tensors, blobs). Any application-level structuring cost exists on *both* transports and is not what Shuttle removes. - -**Reliability / Quality / Portability / Security** - -| ID | Pri | Requirement | -|---|---|---| -| NFR-R1 | M | A peer crash never permanently deadlocks the survivor on Linux (FR-18); macOS degrades to timeout/heartbeat (FR-19). | -| NFR-R2 | M | No named shared-memory object leaks across normal start/stop (verified by inspecting `/dev/shm`). | -| NFR-M1 | M | The multi-process harness passes under AddressSanitizer + ThreadSanitizer with no reported errors. | -| NFR-M2 | S | API behavior, error codes, segment layout, and the memory-ordering contract are documented in one reference. | -| NFR-PO1 | M | Builds and passes its test suite on Linux (primary). | -| NFR-PO2 | S | Builds on macOS (Apple silicon) for development, with the documented robustness caveat. | -| NFR-S1 | M | Named objects are created owner-only by default; wider access is opt-in. | -| NFR-S2 | M | Lengths/offsets read from the header are validated to prevent out-of-bounds access if the segment is corrupted. | - ---- - -## 4. Verification & Benchmark Plan - -### 4.1 Functional verification - -- Unit tests for BipBuffer reserve/commit/release, the A→B switch, and length-framing edge cases (payload exactly filling A; forced early wrap to B; max-size payload). -- Multi-process integration: producer and consumer processes exchange a known sequence; consumer asserts FIFO order and byte-exact content (FR-7, FR-8, FR-9). -- Cross-language: C++↔Python and C++↔Rust round trips (FR-21). -- Crash test: `SIGKILL` the producer while a reservation is in flight; assert the Linux survivor recovers via the robust path (FR-18) and the macOS survivor aborts via heartbeat (FR-19). -- Sanitizers: run the harness under TSan/ASan (NFR-M1). -- Leak check: `/dev/shm` clean after a graceful run (NFR-R2). - -### 4.2 The headline benchmark (D7 → NFR-P1/P2/P3) - -**Two transports, identical workload:** (a) Shuttle; (b) a localhost-HTTP baseline that moves the **raw, uncompressed binary payload as the HTTP body** — no JSON, no base64, keep-alive on, sensible socket/buffer sizes. This keeps the comparison bulletproof: the baseline is HTTP doing the least wasteful thing it can. - -**Workloads:** (1) a single **50 MB** blob (context/tensor case); (2) a stream of small frames (e.g., 16 KB) at a fixed rate (audio case, D5 secondary). - -**Metrics** (warm-up first, then many iterations; report median and p99): end-to-end latency (producer commit → consumer has the full payload); CPU time attributable to copy/serialize per side (profiler); streaming throughput (MB/s and frames/s). - -**Controls:** identical hardware (document the spec — note Linux prod vs. macOS dev figures separately), identical payloads, warm caches. - -**Pass criteria:** NFR-P1 (≥10× on 50 MB), NFR-P2 (negligible copy CPU on the borrow path), NFR-P3 (µs-scale wake). - ---- - -## 5. Appendices - -### Appendix A — Segment layout - -Fixed-width fields; atomics where noted; all internal references are **offsets** (App. B). Pad the header so the data region begins on a cache-line boundary. - -``` -+--------------------------------------------------------------+ offset 0 -| Control Header | -| uint64_t magic | -| uint32_t version, uint32_t flags | -| uint64_t data_capacity | -| -- BipBuffer state (atomics; single-writer ownership) -- | -| atomic a_start (consumer-owned) | -| atomic a_size (producer grows, consumer drains)| -| atomic b_size (producer grows, consumer clears)| -| uint64_t reserve_start, reserve_size (producer-private)| -| -- parking-lot wake -- | -| atomic consumer_waiting, producer_waiting | -| -- liveness (macOS heartbeat) -- | -| atomic producer_heartbeat, consumer_heartbeat | -| pthread_mutex_t lock (PROCESS_SHARED [+ ROBUST on Linux])| -| pthread_cond_t not_empty (PROCESS_SHARED) | -| pthread_cond_t not_full (PROCESS_SHARED) | -+--------------------------------------------------------------+ data_offset -| Data region (BipBuffer): sequence of [u64 len | payload] ... | -+--------------------------------------------------------------+ end -``` - -The memory-ordering contract: producer is the only writer of `a_size`/`b_size`/`reserve_*`; consumer is the only writer of `a_start`. The **A→B handoff** (consumer setting `a_start=0`, adopting `b_size`, clearing `b_size`) is the single most delicate ordering point and must be specified precisely in the design doc with release/acquire fences. - -### Appendix B — Implementation gotchas (read before coding) - -1. **Never store raw pointers in the segment.** Each process maps it at a different base; store offsets, resolve as `base + offset`. -2. **Initialize pshared attributes explicitly** on both the mutex and the condvars; a default-initialized mutex only works within one process. -3. **Robust recovery is real code:** on `EOWNERDEAD`, repair the protected state *then* call `pthread_mutex_consistent` before unlocking, or the mutex is permanently dead (Linux only). -4. **macOS gaps:** no robust mutexes, no `pthread_mutex_timedlock`. Use `pthread_cond_timedwait` + heartbeat. macOS also imposes short `shm_open` name limits and shm size limits — keep names short and check `ftruncate`/`mmap` returns. -5. **Single init:** exactly one process initializes the header; the opener must not proceed until init is published (an init flag/seqlock in the header). -6. **BipBuffer contiguity is the whole point:** never split a payload across the wrap; reserve `8 + len` as one unit and wrap early to B if it won't fit after A. -7. **Lost-wakeup guard:** always re-check the predicate under the mutex after waking; pair the `*_waiting` flag with the signal so the wake path stays off the hot path without dropping wakeups. - -### Appendix C — Platform notes - -- **Linux (prod):** full feature set incl. robust pshared mutexes; objects under `/dev/shm`. Primary, fully-supported target. -- **macOS (dev, Apple silicon):** pshared mutexes yes; robust mutexes and `pthread_mutex_timedlock` no → heartbeat + `cond_timedwait`. Short shm name limits. -- **Windows (future):** different API (`CreateFileMapping`/`MapViewOfFile`, named events); a portability layer is future work. - -### Appendix D — Future work - -Multi-producer/multi-consumer and pub/sub fan-out; a typed schema layer atop the byte transport; a Windows portability layer; optional huge-page backing for very large buffers; an io_uring-style completion API for batched streaming. - ---- - -*End of document.* From 7f03eabe9cf6aff9f9018bbf3096661023ea0b29 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 05:57:45 +0000 Subject: [PATCH 2/5] feat: opt-in transparent huge pages via additive shuttle_create_ex (C ABI v1.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New create-time flag SHUTTLE_CREATE_HUGEPAGES advises the segment mapping with madvise(MADV_HUGEPAGE) on Linux — purely advisory (effective only where the kernel THP shmem policy permits), a no-op on macOS, never a correctness dependency. The flag is persisted in the header's cold identity block before the init_state release-store, so openers advise their own independent mapping too; unknown create-flag bits are masked and never persisted. The frozen v1 surface is untouched: shuttle_create keeps its exact signature and is now the create_flags=0 case of the additive v1.1 entry point shuttle_create_ex (SHUTTLE_ABI_VERSION stays 1). The C flag value is static_asserted against the C++ kFlagHugePages bit. The madvise call lives behind the platform seam in platform.hpp, the only file allowed to ifdef on platform. New test shuttle_hugepage_test (suite now 29): flag persisted and observed by a spawned opener, byte-exact transfer on a flagged channel, plain create leaves the bit clear, unknown bits masked. Asserts the flag contract only — never THP outcomes, which are kernel-policy-dependent and would flake in CI. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01JykzcEronyifSN4RnTgjTk --- CMakeLists.txt | 5 + include/shuttle/header.hpp | 7 ++ include/shuttle/platform.hpp | 17 +++ include/shuttle/shuttle.hpp | 4 +- include/shuttle/shuttle_c.h | 25 ++++- src/shuttle.cpp | 19 +++- src/shuttle_c.cpp | 20 +++- tests/hugepage_test.cpp | 200 +++++++++++++++++++++++++++++++++++ 8 files changed, 286 insertions(+), 11 deletions(-) create mode 100644 tests/hugepage_test.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index a116b48..cdfd8ff 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -145,6 +145,9 @@ target_compile_definitions(shuttle_cabi_errors_test PRIVATE RUST_SRC_DIR="${CMAKE_CURRENT_SOURCE_DIR}/tests/ffi/rust" RUST_OUT_DIR="${CMAKE_CURRENT_BINARY_DIR}") +add_executable(shuttle_hugepage_test tests/hugepage_test.cpp) +target_link_libraries(shuttle_hugepage_test PRIVATE shuttle_c shuttle_core Threads::Threads) + add_executable(shuttle_inspect tools/inspect.cpp) target_link_libraries(shuttle_inspect PRIVATE shuttle_core) @@ -179,6 +182,7 @@ add_test(NAME shuttle_crash_mutex_test COMMAND shuttle_crash_mutex_test) add_test(NAME shuttle_cabi_python_test COMMAND shuttle_cabi_python_test) add_test(NAME shuttle_cabi_rust_test COMMAND shuttle_cabi_rust_test) add_test(NAME shuttle_cabi_errors_test COMMAND shuttle_cabi_errors_test) +add_test(NAME shuttle_hugepage_test COMMAND shuttle_hugepage_test) add_test(NAME shuttle_bench_g71 COMMAND shuttle_bench) add_test(NAME shuttle_nocopy_cpu_test COMMAND shuttle_nocopy_cpu_test) add_test(NAME shuttle_wake_under_load_test COMMAND shuttle_wake_under_load_test) @@ -190,6 +194,7 @@ set_tests_properties(shuttle_spsc_stress_test shuttle_spsc_threads_test shuttle_robust_mutex_test shuttle_crash_leak_test shuttle_crash_mutex_test shuttle_cabi_python_test shuttle_cabi_rust_test shuttle_cabi_errors_test + shuttle_hugepage_test shuttle_bench_g71 shuttle_nocopy_cpu_test shuttle_wake_under_load_test PROPERTIES TIMEOUT 300) diff --git a/include/shuttle/header.hpp b/include/shuttle/header.hpp index 3019e01..de41112 100644 --- a/include/shuttle/header.hpp +++ b/include/shuttle/header.hpp @@ -22,6 +22,13 @@ constexpr uint32_t kVersion = 1; // init_state values: 0 (zero-filled segment) = uninitialized. constexpr uint32_t kInitReady = 0x52454459; // "REDY" +// Create-flag bits for ChannelHeader::flags. Contract: the creator writes the +// full flags word ONCE, in the cold identity block, before the release-store +// that publishes init_state; it is immutable thereafter. Openers must IGNORE +// unknown bits — flags is an additive extension point, so new bits carry no +// kVersion bump (an old opener simply doesn't act on a bit it doesn't know). +constexpr uint32_t kFlagHugePages = 0x1; // creator advised MADV_HUGEPAGE + // Hot atomics get a full line each. 128 B = Apple Silicon line size; also // correct (2x conservative) on x86 (binding minor amendment). constexpr size_t kCacheLine = 128; diff --git a/include/shuttle/platform.hpp b/include/shuttle/platform.hpp index 9366bfd..b565948 100644 --- a/include/shuttle/platform.hpp +++ b/include/shuttle/platform.hpp @@ -17,6 +17,7 @@ #include #include +#include #include #include @@ -50,6 +51,22 @@ inline bool shm_name_ok(const char* name) noexcept { return n >= 2 && n <= kMax; } +// Advise the kernel that a mapping is a good candidate for transparent huge +// pages (opt-in, FR create-flag kFlagHugePages). Purely advisory: on Linux it +// takes effect only where the THP shmem policy permits — e.g. +// /sys/kernel/mm/transparent_hugepage/shmem_enabled set to "advise" or +// "always" — and a kernel that disallows it returns a harmless EINVAL we +// deliberately drop. Never a correctness dependency; a no-op on macOS, which +// has no THP knob. Both creator and opener call this on their own mapping. +inline void advise_huge_pages(void* base, size_t len) noexcept { +#if defined(SHUTTLE_PLATFORM_LINUX) + (void)madvise(base, len, MADV_HUGEPAGE); // result ignored by design +#else + (void)base; + (void)len; +#endif +} + // True where PTHREAD_MUTEX_ROBUST / EOWNERDEAD semantics exist (FR-18). #if defined(SHUTTLE_PLATFORM_LINUX) constexpr bool kHasRobustMutex = true; diff --git a/include/shuttle/shuttle.hpp b/include/shuttle/shuttle.hpp index e85d1ed..acaca20 100644 --- a/include/shuttle/shuttle.hpp +++ b/include/shuttle/shuttle.hpp @@ -36,8 +36,10 @@ struct Channel { // FR-1: shm_open(O_CREAT|O_EXCL) + one-shot ftruncate + mmap + header init, // publishing init last with a release store. Owner-only permissions (NFR-S1). +// create_flags carries opt-in create-time bits (kFlagHugePages); unknown bits +// are masked off. Defaulted for source-compatibility with pre-flags callers. Channel* create(const char* name, size_t capacity_bytes, - size_t max_payload_bytes, int* err); + size_t max_payload_bytes, int* err, uint32_t create_flags = 0); // FR-2/FR-3: attach without re-init; waits for init publication, then // validates magic, version, and header sanity (NFR-S2). diff --git a/include/shuttle/shuttle_c.h b/include/shuttle/shuttle_c.h index c2294ac..2a12d69 100644 --- a/include/shuttle/shuttle_c.h +++ b/include/shuttle/shuttle_c.h @@ -8,12 +8,17 @@ * copy-read returns the non-negative payload length on success). * * Changing any signature, constant, or semantic here is an ABI break and - * requires bumping SHUTTLE_ABI_VERSION. + * requires bumping SHUTTLE_ABI_VERSION. The v1 surface (the 10 functions + * below shuttle_create..shuttle_keepalive) is FROZEN and unchanged; the + * v1.1 additions (shuttle_create_ex + SHUTTLE_CREATE_* below) are strictly + * additive — new symbols only, no existing signature or semantic touched — + * so SHUTTLE_ABI_VERSION stays 1 (old binaries keep linking and running). */ #ifndef SHUTTLE_C_H #define SHUTTLE_C_H #include +#include #ifdef __cplusplus extern "C" { @@ -37,15 +42,29 @@ extern "C" { #define SHUTTLE_ERR_WOULD_BLOCK (-12) #define SHUTTLE_ERR_PEER_DEAD (-13) -/* Flags (IF-3): blocking is the default; OR in SHUTTLE_NONBLOCK for - * try-semantics ("would block" instead of parking). */ +/* Per-op flags (IF-3): blocking is the default; OR in SHUTTLE_NONBLOCK for + * try-semantics ("would block" instead of parking). Passed to the read/write + * entry points. */ #define SHUTTLE_NONBLOCK 0x1 +/* Create-flags (v1.1): a SEPARATE namespace from the per-op flags above — + * these are passed only to shuttle_create_ex's create_flags word, never to + * read/write. Opt-in and additive; unknown bits are masked off by the + * implementation. SHUTTLE_CREATE_HUGEPAGES advises transparent huge pages on + * the segment (advisory; effective only where the kernel THP policy permits). */ +#define SHUTTLE_CREATE_HUGEPAGES 0x1 + typedef struct shuttle_channel shuttle_channel; /* --- lifecycle (FR-1..FR-5) --- */ shuttle_channel* shuttle_create(const char* name, size_t capacity_bytes, size_t max_payload_bytes, int* err); +/* v1.1 additive extension: as shuttle_create, plus a create_flags word + * (SHUTTLE_CREATE_* bits). shuttle_create(name, cap, maxp, err) is exactly + * shuttle_create_ex(name, cap, maxp, 0, err). */ +shuttle_channel* shuttle_create_ex(const char* name, size_t capacity_bytes, + size_t max_payload_bytes, + uint32_t create_flags, int* err); shuttle_channel* shuttle_open(const char* name, int* err); void shuttle_close(shuttle_channel* ch); int shuttle_unlink(const char* name); diff --git a/src/shuttle.cpp b/src/shuttle.cpp index c1e0f47..feb6928 100644 --- a/src/shuttle.cpp +++ b/src/shuttle.cpp @@ -24,7 +24,7 @@ constexpr uint64_t kInitWaitNs = 5ull * 1000000000ull; } // namespace Channel* create(const char* name, size_t capacity_bytes, - size_t max_payload_bytes, int* err) { + size_t max_payload_bytes, int* err, uint32_t create_flags) { if (name == nullptr || name[0] != '/' || capacity_bytes == 0 || max_payload_bytes == 0) { set_err(err, kErrInvalidArgs); @@ -64,12 +64,20 @@ Channel* create(const char* name, size_t capacity_bytes, return nullptr; } + // Opt-in THP: advise the fresh mapping before it is touched. Advisory and + // masked to known bits — an unknown flag must never be persisted (openers + // trust that flags carries only bits they may act on). + const uint32_t flags = create_flags & kFlagHugePages; + if (flags & kFlagHugePages) advise_huge_pages(base, map_len); + // ftruncate zero-fills, so init_state is already 0 (uninitialized) and - // cursors/flags/heartbeats are already 0; set the rest explicitly. + // cursors/heartbeats are already 0; set the rest explicitly. flags is part + // of the cold identity block: written once here, before the init_state + // release-store publish, and immutable after (single-init contract). auto* h = static_cast(base); h->magic = kMagic; h->version = kVersion; - h->flags = 0; + h->flags = flags; h->data_offset = kDataOffset; h->data_capacity = capacity_bytes; h->max_payload = max_payload_bytes; @@ -149,6 +157,11 @@ Channel* open(const char* name, int* err) { return nullptr; } + // The opener's mapping is independent of the creator's; if the creator + // opted into huge pages, advise this mapping too (advisory, ignores + // unknown bits per the flags contract). Only after the header is trusted. + if (h->flags & kFlagHugePages) advise_huge_pages(base, map_len); + set_err(err, kOk); return new Channel{base, map_len, h}; } diff --git a/src/shuttle_c.cpp b/src/shuttle_c.cpp index 5eec4ab..64ac226 100644 --- a/src/shuttle_c.cpp +++ b/src/shuttle_c.cpp @@ -25,6 +25,9 @@ static_assert(SHUTTLE_ERR_CORRUPT == shuttle::kErrCorrupt); static_assert(SHUTTLE_ERR_MSG_TOO_LARGE == shuttle::kErrMsgTooLarge); static_assert(SHUTTLE_ERR_WOULD_BLOCK == shuttle::kErrWouldBlock); static_assert(SHUTTLE_ERR_PEER_DEAD == shuttle::kErrPeerDead); +// Create-flag bits are a separate namespace from the per-op flags, but the C +// value must still track the C++ header bit exactly. +static_assert(SHUTTLE_CREATE_HUGEPAGES == shuttle::kFlagHugePages); struct shuttle_channel { shuttle::Channel* ch = nullptr; @@ -71,12 +74,14 @@ int ensure_borrow(shuttle_channel* h, int flags) { extern "C" { -shuttle_channel* shuttle_create(const char* name, size_t capacity_bytes, - size_t max_payload_bytes, int* err) { +shuttle_channel* shuttle_create_ex(const char* name, size_t capacity_bytes, + size_t max_payload_bytes, + uint32_t create_flags, int* err) { try { int e = 0; - shuttle::Channel* ch = - shuttle::create(name, capacity_bytes, max_payload_bytes, &e); + shuttle::Channel* ch = shuttle::create(name, capacity_bytes, + max_payload_bytes, &e, + create_flags); set_err(err, e); if (ch == nullptr) return nullptr; return new shuttle_channel{ch, nullptr, nullptr, nullptr, 0, false}; @@ -86,6 +91,13 @@ shuttle_channel* shuttle_create(const char* name, size_t capacity_bytes, } } +// Frozen v1 signature: unchanged behavior, now the create_flags=0 case of the +// additive v1.1 entry point. +shuttle_channel* shuttle_create(const char* name, size_t capacity_bytes, + size_t max_payload_bytes, int* err) { + return shuttle_create_ex(name, capacity_bytes, max_payload_bytes, 0, err); +} + shuttle_channel* shuttle_open(const char* name, int* err) { try { int e = 0; diff --git a/tests/hugepage_test.cpp b/tests/hugepage_test.cpp new file mode 100644 index 0000000..d488b14 --- /dev/null +++ b/tests/hugepage_test.cpp @@ -0,0 +1,200 @@ +// Opt-in transparent-huge-page create-flag (kFlagHugePages / the C ABI's +// SHUTTLE_CREATE_HUGEPAGES). Driver exercises the additive v1.1 entry point +// shuttle_create_ex end to end; spawned children open the segment in a +// separate process and observe the persisted flags word. What is asserted is +// the FLAG CONTRACT and byte-exact transport, never a THP outcome: whether the +// kernel actually backs the mapping with huge pages is policy-dependent +// (/sys/kernel/mm/transparent_hugepage/shmem_enabled) and would flake in CI, +// so /proc/self/smaps and THP counters are deliberately untouched. +// +// a. create_ex(HUGEPAGES) succeeds and an opener sees kFlagHugePages set. +// b. a byte-exact payload survives producer->spawned-consumer on that +// huge-page-flagged channel. +// c. plain shuttle_create leaves the flag bit clear. +// d. unknown create-flag bits are masked off (flags shows only known bits) +// and the channel still carries data. +#include + +#include +#include +#include +#include + +#include "proc_util.hpp" +#include "shuttle/shuttle.hpp" // shuttle::open / Channel / kFlagHugePages +#include "shuttle/shuttle_c.h" // the surface under test + +namespace { + +constexpr uint64_t kChildTimeoutNs = 10ull * 1000000000ull; +constexpr size_t kCapacity = 1u << 20; +constexpr size_t kMaxPayload = 1u << 16; +constexpr size_t kPayloadLen = 4096; + +int fail(const char* what, long code) { + std::fprintf(stderr, "FAIL: %s (code=%ld)\n", what, code); + return 1; +} + +// Deterministic, position-dependent pattern so a byte-exact check is meaningful. +unsigned char pattern_byte(size_t i) { + return static_cast((i * 31u + 7u) & 0xFFu); +} + +// Child role: open the segment (C++ header path — an acceptable inspection +// route per spec) and assert flags == the expected word. Openers must observe +// exactly the known bits the creator persisted, never the caller's raw input. +int run_opener(const char* name, uint32_t expected_flags) { + int err = 0; + shuttle::Channel* ch = shuttle::open(name, &err); + if (ch == nullptr) return fail("opener: open()", err); + const uint32_t flags = ch->hdr->flags; + shuttle::close(ch); + if (flags != expected_flags) { + std::fprintf(stderr, "FAIL: opener flags=0x%x want 0x%x\n", flags, + expected_flags); + return 1; + } + return 0; +} + +// Child role: consume exactly one message via the C ABI and verify it byte for +// byte. Lifecycle (unlink) stays with the driver; the consumer only closes. +int run_consumer(const char* name, size_t expected_len) { + int err = 0; + shuttle_channel* ch = shuttle_open(name, &err); + if (ch == nullptr) return fail("consumer: shuttle_open", err); + unsigned char out[kPayloadLen]; + const long n = shuttle_read(ch, out, sizeof out, 0); // blocking + int rc = 0; + if (n < 0 || static_cast(n) != expected_len) { + rc = fail("consumer: read length", n); + } else { + for (size_t i = 0; i < expected_len; ++i) { + if (out[i] != pattern_byte(i)) { + rc = fail("consumer: payload byte mismatch", + static_cast(i)); + break; + } + } + } + shuttle_close(ch); + return rc; +} + +// Driver helper: producer writes the pattern via the C ABI, then a spawned +// consumer reads it back and checks every byte. +int transfer_roundtrip(const char* self, shuttle_channel* ch, + const char* name) { + unsigned char buf[kPayloadLen]; + for (size_t i = 0; i < kPayloadLen; ++i) buf[i] = pattern_byte(i); + // Buffer is empty, so a blocking write completes without parking; the + // message stays queued until the consumer child drains it. + const int wr = shuttle_write(ch, buf, kPayloadLen, 0); + if (wr != SHUTTLE_OK) return fail("driver: shuttle_write", wr); + char len_arg[24]; + std::snprintf(len_arg, sizeof len_arg, "%zu", kPayloadLen); + return shuttle_test::run_child_sync(self, "consumer", name, len_arg, + kChildTimeoutNs); +} + +int run_driver(const char* self) { + const int pid = static_cast(getpid()) % 1000000; + char huge_name[32], plain_name[32], mask_name[32]; + std::snprintf(huge_name, sizeof huge_name, "/shhp.h%d", pid); + std::snprintf(plain_name, sizeof plain_name, "/shhp.p%d", pid); + std::snprintf(mask_name, sizeof mask_name, "/shhp.m%d", pid); + shuttle_unlink(huge_name); // clear any stale objects + shuttle_unlink(plain_name); + shuttle_unlink(mask_name); + + int fails = 0; + int err = 0; + + // (a) create_ex opting into huge pages succeeds; opener sees the bit set. + shuttle_channel* huge = shuttle_create_ex( + huge_name, kCapacity, kMaxPayload, SHUTTLE_CREATE_HUGEPAGES, &err); + if (huge == nullptr) { + ++fails; + fail("create_ex(HUGEPAGES)", err); + } else { + if (shuttle_test::run_child_sync(self, "opener", huge_name, "1", + kChildTimeoutNs) != 0) { + std::fprintf(stderr, "FAIL: huge-page opener saw wrong flags\n"); + ++fails; + } + // (b) byte-exact transfer still works on the flagged channel. + if (transfer_roundtrip(self, huge, huge_name) != 0) { + std::fprintf(stderr, "FAIL: huge-page byte-exact transfer\n"); + ++fails; + } + shuttle_close(huge); + } + + // (c) plain shuttle_create leaves the flag bit clear. + shuttle_channel* plain = + shuttle_create(plain_name, kCapacity, kMaxPayload, &err); + if (plain == nullptr) { + ++fails; + fail("shuttle_create(plain)", err); + } else { + if (shuttle_test::run_child_sync(self, "opener", plain_name, "0", + kChildTimeoutNs) != 0) { + std::fprintf(stderr, "FAIL: plain channel flags not clear\n"); + ++fails; + } + shuttle_close(plain); + } + + // (d) unknown create-flag bits are masked to the known set; channel works. + shuttle_channel* masked = shuttle_create_ex( + mask_name, kCapacity, kMaxPayload, 0xFFFFFFFFu, &err); + if (masked == nullptr) { + ++fails; + fail("create_ex(unknown bits)", err); + } else { + // 0xFFFFFFFF masks down to exactly kFlagHugePages (the one known bit). + if (shuttle_test::run_child_sync(self, "opener", mask_name, "1", + kChildTimeoutNs) != 0) { + std::fprintf(stderr, "FAIL: unknown bits not masked to known set\n"); + ++fails; + } + if (transfer_roundtrip(self, masked, mask_name) != 0) { + std::fprintf(stderr, "FAIL: masked-flags channel transfer\n"); + ++fails; + } + shuttle_close(masked); + } + + if (shuttle_unlink(huge_name) != SHUTTLE_OK || + shuttle_unlink(plain_name) != SHUTTLE_OK || + shuttle_unlink(mask_name) != SHUTTLE_OK) { + std::fprintf(stderr, "FAIL: unlink left an object behind\n"); + ++fails; + } + + if (fails == 0) { + std::printf("hugepage_test ok: create-flag persisted+masked, opener " + "observes it, byte-exact transfer holds (platform=%s)\n", + shuttle::platform_name()); + } + return fails == 0 ? 0 : 1; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc == 1) return run_driver(argv[0]); + if (argc == 4 && std::strcmp(argv[1], "opener") == 0) + return run_opener(argv[2], + static_cast(std::strtoul(argv[3], nullptr, + 0))); + if (argc == 4 && std::strcmp(argv[1], "consumer") == 0) + return run_consumer(argv[2], + static_cast(std::strtoul(argv[3], nullptr, + 0))); + std::fprintf(stderr, + "usage: %s [opener | consumer ]\n", + argv[0]); + return 2; +} From ef3a36e97f9eda47cd25630f46cbfdeb356580e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:01:00 +0000 Subject: [PATCH 3/5] docs: Linux-first quick start, consumer guide, README aligned with new docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lead the quick start with the native Linux CMake path (the production target — CI proves it on ubuntu-24.04), keeping the mac+Docker two-platform harness as the secondary block. New 'Using Shuttle in your project' section: add_subdirectory/FetchContent consumption of shuttle_c (C ABI) or shuttle_core (C++), verified by building a real consumer against the published snippet; vendoring guidance for non-CMake builds; pointer to shuttle_create_ex huge-page opt-in. Verification and Scope sections now stand alone (no links to the deleted dev-process docs); test count updated to 29; repository layout reflects docs/API.md + docs/ROADMAP.md. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01JykzcEronyifSN4RnTgjTk --- README.md | 52 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index e8cc84f..f8b592c 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,17 @@ Shuttle replaces that path for same-host communication: one region of physical R ## Quick start -Requirements: macOS (Apple silicon) with Xcode CLT + CMake, and Docker Desktop for the Linux leg. The FFI tests additionally use `python3` + `cffi` and `rustc` (both preinstalled in the provided container image). +The production target is Linux, and that path needs nothing exotic: a C++17 compiler, CMake (>= 3.25), and the standard build tools. Build and run the full correctness suite under AddressSanitizer/UBSan: + +```sh +cmake -B build -DSHUTTLE_SAN=asan +cmake --build build -j +ctest --test-dir build --output-on-failure +``` + +This is the path CI proves on `ubuntu-24.04` (g++, cmake). The cross-language FFI tests additionally need `python3` + `cffi` and `rustc`. Swap `-DSHUTTLE_SAN=asan` for `-DSHUTTLE_SAN=tsan` to get the ThreadSanitizer build — a separate tree, since ASan and TSan cannot be linked into the same binary. A handful of tests measure latency percentiles and CPU ratios and are only meaningful on quiet, controlled hardware; CI excludes them on shared runners with `ctest -E "bench_g71|park_latency|wake_under_load|nocopy_cpu|trickle"`, and you should too on a busy machine. + +The project also ships a two-platform harness driven by `make`, which builds and runs the same suite natively on macOS (Apple silicon) *and* inside a glibc Linux container under both sanitizers: ```sh make test-mac # native build + full test suite under ASan/UBSan @@ -66,13 +76,42 @@ run_inference(p, len); /* read in place */ shuttle_release_read(ch); ``` +Full function-by-function reference: [docs/API.md](docs/API.md). + The benchmark harness (`shuttle_bench`, built unsanitized at `-O2`) runs all three transports over identical workloads and prints the table above, labeling container runs as virtualized. +## Using Shuttle in your project + +Shuttle is dependency-free (no third-party libraries), so consuming it is deliberately plain. Point a CMake project at this tree with `add_subdirectory` (or `FetchContent`) and link one of two targets: + +- **`shuttle_c`** — the shared library exposing the frozen C ABI (`shuttle/shuttle_c.h`). This is the boundary the Python and Rust bindings link against. +- **`shuttle_core`** — the static library for the C++ API (headers under `include/shuttle/`). + +Both targets declare their headers as `PUBLIC` include directories, so a linking consumer inherits the `include/` path automatically — no manual `target_include_directories` needed: + +```cmake +cmake_minimum_required(VERSION 3.25) +project(my_app LANGUAGES C CXX) + +add_subdirectory(path/to/shuttle shuttle_build) + +add_executable(my_app main.c) +target_link_libraries(my_app PRIVATE shuttle_c) # or shuttle_core for the C++ API +``` + +```c +#include /* resolved via the inherited include path */ +``` + +Because the library has no external dependencies, vendoring `include/` + `src/` straight into your own build is an equally legitimate option; if you go that route (or build without CMake), add `-Iinclude` and compile `src/shuttle.cpp` (plus `src/shuttle_c.cpp` for the C ABI) yourself. + +For opt-in transparent huge pages on the segment, create the channel with `shuttle_create_ex(name, cap, maxp, SHUTTLE_CREATE_HUGEPAGES, &err)` — an additive C ABI v1.1 symbol (advisory `madvise` on Linux, no-op elsewhere); see [docs/API.md](docs/API.md). + ## Verification -The build was driven gate-by-gate through an 8-phase plan ([docs/Shuttle_Implementation_Plan.md](docs/Shuttle_Implementation_Plan.md)) with one rule: **one new variable per phase** — data-structure logic proven before concurrency, concurrency before IPC, ordering before wake mechanics, wake before crash recovery. All 27 gates passed on both platforms; the complete ledger with per-gate evidence, dated decisions, and the failures encountered along the way is in [PROGRESS.md](PROGRESS.md). +The build was driven gate-by-gate with one rule: **one new variable per phase** — data-structure logic proven before concurrency, concurrency before IPC, ordering before wake mechanics, wake before crash recovery. The test suite is the standing evidence, and it stands alone. -Highlights of what the suite (28 tests, ASan + TSan clean on both legs) actually proves: +Highlights of what the suite (29 tests, ASan + TSan clean on both legs) actually proves: - 200k-pair randomized property test of the BipBuffer with invariants checked after every operation (19k+ wraps in the tight configuration). - ≥1 GiB two-process byte-exact FIFO stress; asymmetric-speed stress with the spin paths *proven engaged*; a wrap-heavy stress that fires the delicate A→B handoff 57k times. @@ -89,7 +128,7 @@ Highlights of what the suite (28 tests, ASan + TSan clean on both legs) actually ## Scope (v1.0) -Same-host, single-producer/single-consumer, one-way channels. Cross-machine transport, Windows, MPMC/pub-sub, and payload schemas are explicitly out of scope (see [docs/Shuttle_SRS.md](docs/Shuttle_SRS.md)). macOS crash recovery is best-effort by design (no robust mutexes exist there); Linux is the hard-guarantee platform. +Same-host, single-producer/single-consumer, one-way channels. Cross-machine transport, Windows, MPMC/pub-sub, and payload schemas are explicitly out of scope. macOS crash recovery is best-effort by design (no robust mutexes exist there); Linux is the hard-guarantee platform. Proposed directions past v1 — and why each stays out of it — are triaged in [docs/ROADMAP.md](docs/ROADMAP.md). ## Repository layout @@ -98,8 +137,7 @@ include/shuttle/ header.hpp (segment layout), bipbuffer.hpp (core logic), spsc.hpp (lock-free path + parking), platform.hpp (the ONLY file allowed to #ifdef on platform), shuttle_c.h (C ABI v1) src/ lifecycle (shm_open/mmap/validate) + C ABI implementation -tests/ 28 gate tests; tests/ffi/ holds the Python + Rust bindings +tests/ 29 gate tests; tests/ffi/ holds the Python + Rust bindings bench/ three-transport benchmark harness -docs/ SRS, implementation plan, build directive -PROGRESS.md the complete build ledger: every gate, decision, and dead end +docs/ API.md (frozen C ABI reference), ROADMAP.md (post-v1 triage) ``` From 1223f8754f2e141fe40eb320b76249a74d6f016c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:07:54 +0000 Subject: [PATCH 4/5] ci: publish a GitHub Release on version-tag push Tag pushes matching v* create a release whose notes come from the annotated tag message (--notes-from-tag), so release notes are curated at tag time and versioned in git. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01JykzcEronyifSN4RnTgjTk --- .github/workflows/release.yml | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 .github/workflows/release.yml diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..3fa717f --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,25 @@ +# Publish a GitHub Release when a version tag (v*) is pushed. The annotated +# tag's message becomes the release notes (--notes-from-tag), so notes are +# curated at tag time and live in git history, not in this file. +name: Release + +on: + push: + tags: ["v*"] + +permissions: + contents: write + +jobs: + release: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + - name: Create GitHub release from tag annotation + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release create "$GITHUB_REF_NAME" \ + --verify-tag \ + --title "Shuttle $GITHUB_REF_NAME" \ + --notes-from-tag From 5135048a1891bccf54bb2b0393be20cca9ecec2c Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 24 Jul 2026 06:16:44 +0000 Subject: [PATCH 5/5] ci: dispatchable release path with versioned notes; v1.0.0 notes Add a workflow_dispatch trigger to the Release workflow: it creates the tag at the dispatched ref and publishes the release, taking notes from .github/releases/.md when present. Needed because this environment can push branches but not tags; the tag-push path remains for local tagging. Ship the v1.0.0 notes file. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01JykzcEronyifSN4RnTgjTk --- .github/releases/v1.0.0.md | 22 +++++++++++++++++++++ .github/workflows/release.yml | 37 +++++++++++++++++++++++++++-------- 2 files changed, 51 insertions(+), 8 deletions(-) create mode 100644 .github/releases/v1.0.0.md diff --git a/.github/releases/v1.0.0.md b/.github/releases/v1.0.0.md new file mode 100644 index 0000000..68254ae --- /dev/null +++ b/.github/releases/v1.0.0.md @@ -0,0 +1,22 @@ +First tagged release: a zero-copy shared-memory IPC framework for local AI workloads. C++17 core, lock-free SPSC data path, crash-resilient, with a frozen C ABI driven from Python and Rust. + +## Highlights + +- **Bipartite buffer (BipBuffer) ring**: every payload is physically contiguous, so the zero-copy pointer handoff is always valid. +- **Lock-free hot path**: single-writer atomic cursors published with release/acquire ordering; zero locks taken when the peer isn't parked. +- **Parking, not polling**: blocked peers sleep at ~0.05% CPU and wake in microseconds; every wait is a bounded timedwait. +- **Crash resilience**: heartbeat liveness on both platforms plus robust-mutex (`EOWNERDEAD`) recovery on Linux; a peer SIGKILLed mid-transfer leaves the survivor with a clean `PEER_DEAD` error, never a deadlock. +- **Frozen C ABI v1** with Python (cffi, zero-copy `memoryview`) and Rust (compile-time borrow enforcement) bindings. +- **New in this release**: opt-in transparent huge pages via the additive `shuttle_create_ex` + `SHUTTLE_CREATE_HUGEPAGES` (advisory `madvise` on Linux, no-op elsewhere; the frozen v1 surface is untouched). + +## Verification + +29-test suite green under AddressSanitizer and ThreadSanitizer on macOS (native Apple silicon) and Linux (glibc); the correctness suite runs in CI on ubuntu-24.04. Consolidated API reference in [docs/API.md](https://github.com/allandng/shuttle/blob/main/docs/API.md); direction triage in [docs/ROADMAP.md](https://github.com/allandng/shuttle/blob/main/docs/ROADMAP.md). + +## Benchmark honesty + +Headline figures (50 MB end-to-end in ~5 µs vs ~9 ms over Unix sockets) are development measurements on native Apple M-series hardware. The production target is Linux and the headline claim remains provisional until the harness runs on bare-metal Linux; see the README's "Benchmark honesty" section for methodology and caveats. + +## Scope + +Same-host, single-producer/single-consumer, one-way channels. Linux is the hard-guarantee platform; macOS crash recovery is best-effort by design. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3fa717f..5bc7651 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,11 +1,21 @@ -# Publish a GitHub Release when a version tag (v*) is pushed. The annotated -# tag's message becomes the release notes (--notes-from-tag), so notes are -# curated at tag time and live in git history, not in this file. +# Publish a GitHub Release. Two triggers: +# - push of a version tag (v*): the annotated tag's message becomes the +# release notes (--notes-from-tag), curated at tag time. +# - workflow_dispatch with a tag input: creates the tag at the dispatched +# ref and publishes the release, taking notes from +# .github/releases/.md when that file exists (falling back to +# GitHub's generated notes). This path exists for environments that can +# push branches but not tags. name: Release on: push: tags: ["v*"] + workflow_dispatch: + inputs: + tag: + description: "Tag to release (created at the dispatched ref if it does not exist)" + required: true permissions: contents: write @@ -15,11 +25,22 @@ jobs: runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 - - name: Create GitHub release from tag annotation + - name: Create GitHub release env: GH_TOKEN: ${{ github.token }} + EVENT_NAME: ${{ github.event_name }} + INPUT_TAG: ${{ inputs.tag }} run: | - gh release create "$GITHUB_REF_NAME" \ - --verify-tag \ - --title "Shuttle $GITHUB_REF_NAME" \ - --notes-from-tag + if [ "$EVENT_NAME" = "workflow_dispatch" ]; then + NOTES=".github/releases/${INPUT_TAG}.md" + if [ -f "$NOTES" ]; then + gh release create "$INPUT_TAG" --target "$GITHUB_SHA" \ + --title "Shuttle $INPUT_TAG" --notes-file "$NOTES" + else + gh release create "$INPUT_TAG" --target "$GITHUB_SHA" \ + --title "Shuttle $INPUT_TAG" --generate-notes + fi + else + gh release create "$GITHUB_REF_NAME" --verify-tag \ + --title "Shuttle $GITHUB_REF_NAME" --notes-from-tag + fi