This repository was archived by the owner on Jan 19, 2026. It is now read-only.
Entirely new spec, re-impl for Rust, Swift, TypeScript, Java and Go#150
Closed
fasterthanlime wants to merge 69 commits into
Closed
Entirely new spec, re-impl for Rust, Swift, TypeScript, Java and Go#150fasterthanlime wants to merge 69 commits into
fasterthanlime wants to merge 69 commits into
Conversation
- Rewrote message format: Message struct with channel_id + MessagePayload enum - Each message type only carries fields it needs (no fixed 64-byte descriptor) - Added Channels section explaining Call/Stream/Tunnel kinds - Added Method Identity section with blake3 hash computation - Added Introspection section with optional Diagnostic service - Reorganized Transports: message, multi-stream, byte-stream (COBS framing) - Moved SHM to separate spec document
- Main spec now lean: just protocol concepts, no signature hash details - New rust-spec/ with signature hash algorithm, TypeDetail, introspection types - Other languages get pre-computed method IDs from codegen - Updated Method Identity to reference Rust spec for sig_bytes computation
The method_id computation algorithm (blake3 hash with kebab-case) is Rust-specific implementation detail, not a language-agnostic protocol requirement. Other language implementations generate their code from Rust proto crates, so they don't need to compute method IDs themselves. - Remove r[method.identity.computation] rule from main spec - Keep high-level method identity concepts in main spec - Add schema evolution section documenting breaking changes - Clarify that Rapace is explicitly Rust-native (not language-neutral) - Add rust-spec and shm-spec to tracey config
The previous encoding was ambiguous — fn add(a: i32, b: i32) -> i64 would produce the same bytes as fn foo(a: i32, b: i32, c: i64) returning unit. Now method signatures use the Tuple tag (0x25) with an explicit argument count, making parsing unambiguous.
Comprehensive Unary RPC specification: - Request ID scoping: tracked by (channel_id, request_id) tuple - Request ID uniqueness: never reuse within a channel's lifetime - Duplicate detection: best-effort, cite rule ID in Goodbye on violation - In-flight definition: until Response received (streams live independently) - Request/Response message fields and payload encoding - Call errors vs application errors distinction - Call lifecycle with single-response guarantee - Cancellation semantics (best-effort, callee still responds) - Pipelining: multiple in-flight requests, independent processing Also adds Goodbye message for graceful protocol error handling.
- Add r[unary.request-id.cancel-still-in-flight] rule: cancellation doesn't release request IDs until Response arrives - Add request state diagram (Idle <-> In-Flight) showing Cancel as a self-loop with no state change - Replace CallError with RapaceError<E> which distinguishes User(E) application errors from protocol errors (UnknownMethod, etc.) - Define wire discriminants explicitly in main spec (0=User, 1-5=protocol) - Add inline type boxes for Request, Response, Cancel messages - Add Stream<T> -> u64 wire encoding to Rust spec - Rust spec now references main spec for RapaceError definition
- Add credit-based flow control for streams (per-stream, byte-based) - Add Credit message type for granting stream credits - Add metadata field to Request/Response with MetadataValue enum - Rename Eos to Close, CloseStream to Reset for clarity - Remove Timeout and Internal from RapaceError (now 4 variants) - Make multistream transport rules normative (MUST not SHOULD) - Simplify Introspection section (point to rapace-discovery crate) - Fix various MUST NOT rules to be receiver-side checks instead
- Define Hello message as enum (Hello::V1) for versioning - Add max_payload_size and initial_stream_credit to Hello - Specify Goodbye receive behavior (stop sending, close, fail in-flight) - Remove Ping/Pong (transports have their own keepalive) - Reserve stream ID 0 (connection error if used) - Specify Reset behavior: ignore further Data/Close, credit is lost - Add metadata limits: max 128 keys, max 1MB per value
- Add References section with [POSTCARD], [RUST-SPEC], [SHM-SPEC], [COBS] - Update inline references to use citation labels - Add explicit rules for UnknownMethod and InvalidPayload errors - Change request IDs to monotonically increasing counter (no reuse) - Remove state diagram (Idle/In-Flight cycle is implicit) - Simplify Schema Evolution section to 3 bullet points - Clarify that call errors don't close the connection
Main spec changes: - Ban Stream<T> in error types (r[streaming.error-no-streams]) - Clarify Data payload: exactly one T per message - Add Credit to ignored messages after Reset - Add Hello version mismatch rule and message ordering requirement - Rewrite multi-stream transport section: - Rapace streams are unidirectional - Dedicated streams use COBS-framed raw T values (no Message wrapper) - Close/Reset use transport stream semantics (FIN/RESET_STREAM) - Credit goes on control stream 0 - Goodbye reason MUST contain rule ID (e.g., streaming.id.zero-reserved) - Fix argument names: not part of wire format, can be renamed freely Rust spec changes: - Add r[wire.stream.not-in-errors] requiring compile-time check
- Add explicit byte accounting: count Postcard encoding length, not framing - Clarify credit applies to ALL transports (multi-stream included) - Rapace credit operates independently of transport-level flow control - Add sender-side obligation: MUST NOT send if would exceed credit - Add r[flow.stream.credit-prompt]: Credit must be processed promptly - Add r[flow.stream.zero-credit]: waiting is not a protocol error - Add non-normative implementation guidance: - When to grant credits (after consume vs after buffer) - Hysteresis pattern (grant at W/2 to restore to W) - Timeout advice for zero-credit stalls - Clarify Credit.stream_id maps to transport stream on multi-stream
- Byte accounting: clarify Data.payload applies to all message-based
transports, multi-stream writes bytes 'before COBS framing'
- Downgrade credit-prompt from MUST to SHOULD, define in non-timing
terms ('without intentional delay')
- Add r[transport.multistream.stream-id-mapping]: Rapace stream_id
MUST equal transport stream ID (enables interop without negotiation)
- Update credit-grant to reference the new mapping rule
Core vs transport boundary: - Add 'Specification Scope' section clarifying core semantics vs transport bindings - Core: service definitions, RPC lifecycle, stream semantics, flow control - Transport bindings: framing, Hello/Goodbye, stream ID allocation Stream directionality fix: - Streams are bidirectional with half-close (consistent with core definition) - Multi-stream transports map to bidirectional transport streams - Removed incorrect 'unidirectional' claim Stream ID mapping fix: - Removed 'stream_id MUST equal transport stream ID' (too restrictive) - Transport stream IDs (QUIC) may differ from Rapace stream IDs - Rapace stream_id is allocated per r[streaming.id.parity] - Implementation maintains local mapping to transport stream handles - Removed stale reference from flow.stream.credit-grant
Major restructuring to separate core semantics from transport bindings: Core Semantics (transport-agnostic): - r[core.call]: Request/Response lifecycle, request_id uniqueness - r[core.stream]: Streams are UNIDIRECTIONAL (sender → receiver) - r[core.stream.close]: Close ends the stream (no half-close) - r[core.stream.reset]: Either peer can reset - r[core.stream.id.*]: Unique IDs, zero reserved, disjoint allocation - r[core.error.*]: RapaceError variants, call vs connection errors - r[core.flow.*]: Credit-based flow control principles - r[core.metadata]: Key-value pairs, unknown keys ignored Key design decisions: - Streams are unidirectional: bidirectional = two streams - This aligns with QUIC/WebTransport unidirectional streams - Removes half-close complexity from core - Stream ID allocation scheme is binding-specific (core just says 'disjoint') Updated transport bindings: - Multi-stream: map to unidirectional transport streams - Removed stale 'half-close' references - Close = QUIC FIN (stream done), Reset = QUIC RESET_STREAM This sets up clean separation for SHM-SPEC to define its own binding while following core semantics.
Complete SHM binding to Core Semantics: Scope: - Encodes Core messages over shared memory - Does not redefine call/stream/error semantics Topology: - Pair (peer-to-peer): creator odd IDs, attacher even IDs - Hub (1:N): deferred to future version Segment Layout: - 128-byte header with magic, version, sizes, offsets - SPSC descriptor rings (one per direction) - Payload slot region with generation counters Message Encoding: - 64-byte MsgDesc descriptors (cache-line aligned) - msg_type field maps to Core abstract messages - Inline payload (≤32 bytes) or slot-based - NO Credit message type (uses shared counters) Flow Control (Option 2 - shared counters): - granted_total: monotonic AtomicU32 published by receiver - remaining credit = granted_total - sent_total (wrapping) - Release ordering on grant, Acquire on send check - Futex/condvar for zero-credit waiting - Counters ignored after Reset Ordering and Synchronization: - Ring: Release on enqueue, Acquire on dequeue - Payload visibility guaranteed by ordering - Futex for wakeups (fallback: polling, platform primitives) Failure: - Goodbye flags in segment header - Optional Goodbye descriptor with reason - Epoch counters for crash detection Byte Accounting: - payload_len of Data descriptors (POSTCARD-encoded size) - Consistent with r[core.flow.byte-accounting]
Remove Pair topology entirely. Focus on 1:N hub topology for plugin systems (DotaCast use case). Changes: - Hub topology: one host, multiple guests - Per-guest rings (guest→host, host→guest) - Per-guest slot pools - Stream ID encoding with peer_id in upper bits (31-24) - Shared counter flow control (no Credit message type) - Guest lifecycle (attach/detach) with epoch-based crash detection - Peer table with per-guest state tracking
Fixes based on review feedback: 1. ID width mismatch: Explicitly document that SHM uses u32 for request_id and stream_id (upper 32 bits implicitly zero) 2. peer_id: Now u8 (index into peer table), max_guests capped at 255 3. Stream ID scope: Clarified as pair-scoped (not globally unique), removed contradictory bit-packing scheme 4. peer_id assignment: Explicitly defined as peer table index 5. Metadata encoding: Added - metadata is part of Request/Response payload, POSTCARD-encoded with arguments/result 6. Credit counter location: Added StreamEntry struct and stream metadata table with concrete layout and indexing rules 7. Crash detection: Added host-owned detection requirement with heartbeat option, detailed recovery steps 8. Host slot pool: Specified location at slot_region_offset, before per-guest pools Also added: - stream_table_offset to PeerEntry - last_heartbeat field for crash detection - heartbeat_interval to segment header - max_streams to segment header
Critical fixes (spec-breaking): 1. peer_id contradiction: peer_id is now 1 + peer_table_index (so guests are 1-255, host has no peer_id). Clarified table has exactly max_guests entries. 2. Stream ID collision: Added odd/even parity within guest-host pair. Host uses even IDs (2, 4, 6...), guest uses odd (1, 3, 5...). 3. Stream table indexing: stream_id directly indexes table (no modulo). Stream IDs must be < max_streams. Added stream ID reuse protocol. Additional clarifications: 4. Metadata encoding: Added note explaining why SHM combines metadata and arguments/result (64-byte descriptor constraint). 5. Heartbeat clock domain: Specified tick counts using monotonic clock (CLOCK_MONOTONIC on Linux, QPC on Windows, mach_absolute_time on macOS). 6. Slot allocation/free: Added SlotPoolHeader with free_bitmap, atomic CAS allocation, atomic OR for freeing, generation counter location (first 4 bytes of each slot). 7. Ring-full semantics: Added ring capacity rule (ring_size - 1 usable), specified producer waits on full ring (futex on tail).
fasterthanlime
changed the title
fasterthanlime
changed the title
…ompat - Fix redundant guard in rapace-hash kebab conversion - Remove legacy rapace-core references from xtask (doc, clippy, coverage) - Drop Swift tools version to 5.9 for CI compatibility - Formatting pass on all rust/* and spec/* crates - Add rapace-reflect crate (facet-based TypeDetail reflection)
- Add ServiceDetail struct to group methods by service - Add doc field to MethodDetail - TypeScript: generate_service() produces client/server interfaces - Python: generate_service() produces Protocol/ABC classes + dispatcher - Update tracey config with all 6 language implementations - Add Echo service fixture for testing
- Use heck crate for case conversion (ToUpperCamelCase, ToLowerCamelCase, ToSnakeCase) instead of hand-rolled functions - Add generate_service() for TypeScript, Python, Swift, Go, and Java - Each generates client interface, server handler, and dispatcher - Add r[verify ...] annotations to conformance tests for tracey tracking - Update tracey config to include spec/** for verification detection - All 10 codegen tests passing Tracey now shows 5% verification coverage (5/103 rules): - message.hello.timing - message.hello.ordering - message.hello.unknown-version - flow.unary.payload-limit - streaming.id.zero-reserved
Create rust/rapace/ as the single dependency users need. The #[service] macro now generates code that references rapace::session, rapace::schema, rapace::hash, rapace::reflect, and rapace::__private::facet_postcard. This uses crate name detection (via facet-cargo-toml) to handle cases where users rename the rapace dependency in their Cargo.toml. - Add rust/rapace/ facade crate with re-exports - Add crate_name.rs to rapace-macros for dependency detection - Update all generated code to use rapace::* paths - Update spec-proto and subject-rust to use rapace - Clean up workspace: remove legacy members and demos
- Replaced manual TokenStream2 parsing with structured Type enum - Added Type::has_lifetime(), Type::as_result(), Type::contains_stream() - Removed type_info.rs module entirely - Implemented proper recursive descent type parser using unsynn patterns feat(reflect): add runtime validation for streaming.error-no-streams - Added TypeDetail::visit() visitor pattern in rapace-schema - Implemented Stream detection in error types using structural comparison - Validation happens at runtime during facet Shape processing - Added verification tests with r[verify streaming.error-no-streams] All macro and reflect tests pass. No compile-time validation - spec requires runtime validation via Facet Shapes as noted in comment.
Created Swift core runtime following Rust/TypeScript architecture pattern. Core runtime components (rapace-runtime package): - Binary encoding: Varint (LEB128), COBS framing - Postcard helpers: string/bytes encoding, Result encoding - UnaryDispatcher<Service>: server-side dispatch with method handler map - UnaryCaller protocol: client-side interface for making calls - Error types: RapaceError, CallError<UserError> All tests pass (5 tests: varint encoding, COBS roundtrip, etc.) Codegen should generate only: - Method IDs (computed from signatures) - Method handler map for dispatcher - Client stubs using UnaryCaller - Service-specific encoding/decoding This achieves the same clean separation as Rust (rapace-session) and TypeScript (@bearcove/rapace-runtime) where runtime handles protocol mechanics and codegen only produces service-specific thin wrappers.
**Rust fixes:** - Remove unused `ToTokens` import from parser.rs - Add `ToTokens` import to test module for `.to_token_stream()` calls - Update test assertions to use `.to_token_stream().to_string()` **TypeScript fixes:** - Fix index.ts: import functions before re-exporting for local use - Remove parameter property syntax (not supported by Node's experimental TS stripper) - Use `import type` for interface imports in subject.ts **Test results:** - ✅ All Rust tests pass (macros + conformance) - ✅ All TypeScript tests pass (7/7) - ❌ Swift/Go/Java/Python still failing (pre-existing, unrelated to parser changes)
**Swift code generation:** - Wire up Swift codegen to xtask CLI with `--swift` flag - Implement complete dispatcher generation in swift.rs: - Decode request payload (postcard tuple of arguments) - Call handler with proper Swift argument labels - Encode response as Result<T, RapaceError<E>> - Handle UnknownMethod, InvalidPayload errors per spec - Add RapaceRuntime import to generated code **Swift subject implementation:** - Replace hand-coded protocol parsing with generated dispatcher - Implement EchoService handler (echo, reverse methods) - Add stream_id=0 validation (r[streaming.id.zero-reserved]) - Send Goodbye on protocol errors per spec **Test results:** - ✅ All 7 Swift conformance tests pass - ✅ unary_echo_roundtrip - ✅ unary_unknown_method_returns_unknownmethod_error - ✅ stream_id_zero_triggers_goodbye - ✅ All protocol handshake tests This brings Swift to parity with TypeScript and Rust implementations.
Implement complete dispatcher generation for Go, Java, and Python, matching the Swift pattern. All three languages now pass 7/7 conformance tests. Changes: - Add codegen CLI flags (--go, --java, --python) to xtask - Implement dispatcher generation in each target: - Decode request payloads (postcard-encoded arguments) - Call handler methods - Encode responses as Result<T, RapaceError<E>> - Return proper error codes (UnknownMethod, InvalidPayload) - Include runtime helpers in generated code - Update subjects to use generated dispatchers - Create Echo service implementations for each language Test results: - Go: 7/7 passing - Java: 7/7 passing - Python: 7/7 passing
Contributor
Author
|
Gonna close this and continue it in another repository. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the third iteration of rapace. The first one was a Rust implementation with two other implementations tackled on top without a real spec. The second one was way over-specified. The spec was too long and impossible to implement and test fully.
This is the third implementation, which strives to be somewhere in the middle. The implementation for all five languages is rebuilt from the beginning using tracey to track conformance and coverage and everything. Hopefully this time we end up with five compatible implementations and a spec that makes sense to humans.