RustBinary is a bounded binary codec for Serde with explicit wire profiles and opt-in systems for adaptive encoding, bit packing, schema identity, CBOR, compression, authenticated encryption, schema evolution, and parallel batches.
The product surface is deliberately split into three layers:
| Layer | Public surface | Default | Scope |
|---|---|---|---|
| Core | rustbinary::core |
Yes | Compact V1 encode/decode, limits, trailing policy, deterministic primitive encoding, caller buffers, no_std |
| Protocol | rustbinary::protocol |
No | evolution, fingerprints, reflection, static bounds, bit packing, compatibility profiles |
| Pipeline | rustbinary::pipeline |
No | CBOR, compression, encryption, ordered parallel transforms |
Read-only memory-mapped object storage is an independent, opt-in
rustbinary::archive surface. It is not enabled by the Core, Protocol, or
Pipeline bundles and does not change their wire formats or no_std boundary.
RustBinary is organized around its own explicit format model:
- Wire-format changes are visible in the configuration type chain. Enabling a Cargo feature never silently rewrites ordinary fields.
- Adaptive strategies compare complete encoded costs, including tags and lengths, and use stable tie-breaking rules.
- Caller-owned memory is an API contract, not an optimizer accident.
- Resource limits and trailing policy propagate through format wrappers.
- Compatibility hashing, deterministic encoding, compression, and AEAD have separate responsibilities and are not presented as interchangeable safety mechanisms.
This produces a canonical data-aware frame family, a stable-field-ID evolution format, and a typed transform pipeline. Compatibility with a different binary format is not implied unless a profile explicitly documents it.
| Capability | Status | Implementation |
|---|---|---|
| Binary Serde codec | Implemented | Fixed-width compatibility and strict marker-varint profiles |
| Adaptive integers | Implemented | Per-value marker width and ZigZag signed values |
| Adaptive strings | Implemented | Runtime choice between raw UTF-8 and ASCII7 |
| Adaptive collections | Implemented | Runtime minimum-size choice among raw, delta, and RLE i64 frames |
| SIMD | Implemented for hot scans | Runtime SSE2/AVX2 on x86_64 and NEON on AArch64; scalar fallback |
| AVX-512, SVE, SME | Detection only | Reported by hardware_capabilities; no codec kernels are claimed |
| Zero-allocation codec paths | Implemented | Exact-size Serde output and caller-owned adaptive decode buffers |
| Borrowed zero-copy decoding | Implemented | Nested &str and &[u8] point directly into the input frame |
| Read-only relative-pointer archive | Implemented behind archive |
Versioned envelope, explicit schema ID, bounded validation, and in-place mmap access |
| Bit packing | Implemented | BitPacked derive, checked widths, canonical zero padding |
| Schema fingerprinting | Implemented | Type structure plus complete binary/CBOR configuration |
| Compile-time bounds | Implemented | StaticSize::{MAX_SIZE, PACKED_MAX_BITS, PACKED_MAX_SIZE} |
| RFC 8949 CBOR | Implemented | Ciborium codec plus recursive canonical map ordering mode |
| Schema evolution | Implemented | Stable field IDs, versions, defaults, unknown-field skipping, migrations |
| Compression | Implemented | Adaptive Zstandard frame; raw data retained when compression loses |
| Encryption | Implemented | XChaCha20-Poly1305, random 192-bit nonce, authenticated frame header |
| Deterministic encoding | Implemented in explicit modes | Bit-packed output, schema frames, parallel batches, deterministic CBOR |
| Parallel serialization | Implemented | Ordered scoped-worker batch frames |
| Runtime reflection | Implemented | Allocation-free static metadata generated by Reflect |
std::io streams |
Implemented | Reader/writer APIs live in adapters with resource limits |
no_std |
Implemented | Compact V1 slice encode/decode and caller-owned buffers require no default features |
no_std + alloc |
Implemented | Vec, String, owned values, fingerprinting, evolution, and scalar adaptive codecs |
| Async fiber/UFA | Not implemented | No fake async wrapper is exposed over blocking I/O |
The distinction is deliberate: hardware detection is not described as hardware acceleration, and the Serde codec and relative-pointer archive remain separate formats and APIs.
[dependencies]
rustbinary = "0.1.4"
serde = { version = "1", features = ["derive"] }Enable a complete optional layer only when it is actually needed:
rustbinary = { version = "0.1.4", features = ["protocol"] }
# or select only the exact capability:
rustbinary = { version = "0.1.4", features = ["fingerprint", "derive"] }
# or select immutable memory-mapped archives without the other layers:
rustbinary = { version = "0.1.4", features = ["archive"] }The minimum supported Rust version is declared in Cargo.toml. Optional
systems only compile when their feature is enabled.
RustBinary requires Rust 1.87 or newer and uses Rust 2021 edition. The optional Zstandard dependency requires a platform C toolchain.
| Feature | Default | Purpose and dependency |
|---|---|---|
std |
Yes | Owned Core and I/O APIs; required by Pipeline and runtime SIMD features |
alloc |
Via std |
Owned Vec/String APIs without requiring std |
protocol |
No | Complete Protocol layer convenience bundle |
pipeline |
No | Complete Pipeline layer convenience bundle |
archive |
No | Validated read-only mmap archives; requires std, rkyv, and memmap2 |
derive |
No | Re-exports procedural macros selected with their runtime feature |
fingerprint |
No | Structural fingerprint runtime and frames |
reflection |
No | Allocation-free reflection runtime |
static-size |
No | Compile-time bounds runtime |
simd |
No | Runtime detection and hot-scan dispatch; never changes bytes |
bit-packing |
No | Core bit-level traits and caller-buffer codec |
adaptive |
No | Caller-buffer adaptive strings/collections; implies bit-packing; alloc adds owned APIs |
cbor |
No | RFC 8949 through Ciborium |
compression |
No | Adaptive Zstandard frame |
encryption |
No | XChaCha20-Poly1305, OS randomness, zeroization |
parallel |
No | Scoped-thread ordered batch frames |
schema-evolution |
No | Stable-field-ID versioned frames |
The primary architecture is RustBinary Compact V1: a pure no_std slice core,
an alloc extension for owned data, and std adapters for streams and platform
services.
cargo build --no-default-features
cargo build --no-default-features --features alloc
cargo build --features stdThe top-level serialize and deserialize functions and options() use the
strict compact profile: little endian, canonical marker varints, ZigZag signed
integers, a 64 MiB byte limit, a 1,000,000-element collection limit, and
rejected trailing bytes. legacy_options() explicitly selects the former
unbounded fixed-width profile and allowed trailing bytes.
use serde::{Deserialize, Serialize};
#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Packet {
sequence: u64,
payload: Vec<u8>,
}
let config = rustbinary::options()
.with_varint_encoding()
.with_little_endian()
.with_limit(8 * 1024 * 1024)
.with_collection_limit(100_000)
.reject_trailing_bytes();
let packet = Packet { sequence: 42, payload: vec![1, 2, 3] };
let bytes = config.serialize(&packet)?;
assert_eq!(config.deserialize::<Packet>(&bytes)?, packet);
# Ok::<(), rustbinary::Error>(())Configuration values are small and copyable. Format-changing methods return a different wrapper, making transform order visible at compile time:
Config -> CborConfig -> CompressedConfig -> EncryptedConfig
The allowed method order is the allowed processing order. Encryption cannot accidentally run before compression through this chain.
The format encodes values, never Rust object memory. It does not serialize
padding, native pointers, vtables, or repr(Rust) layout.
| Serde value | Wire representation |
|---|---|
bool |
One byte, exactly 0 or 1 |
Option<T> |
One-byte 0/1 tag, then T for Some |
u8 / i8 |
One byte |
| Fixed integer | Exact declared width in configured endian |
| Variable unsigned integer | Canonical marker plus 0/2/4/8/16 payload bytes |
| Variable signed integer | ZigZag followed by unsigned marker encoding |
f32 / f64 |
IEEE 754 bits in configured endian |
char |
One valid UTF-8 scalar without a length prefix |
| String / bytes | Encoded byte length followed by exact bytes |
| Sequence / map | Declared count followed by values or entries |
| Tuple / struct | Fields in Serde declaration order without names |
| Enum | Configured u32 variant index followed by variant data |
Marker varints are canonical:
| Marker | Payload | Minimum accepted value |
|---|---|---|
0..=250 |
None; marker is the value | 0 |
251 |
2 bytes | 251 |
252 |
4 bytes | 65,536 |
253 |
8 bytes | 4,294,967,296 |
254 |
16 bytes | 18,446,744,073,709,551,616 |
255 |
Reserved and invalid | Never accepted |
Decoders reject non-minimal forms, narrowing overflow, malformed UTF-8, invalid primitive tags, truncation, resource-limit violations, and disallowed trailing bytes.
serialized_size uses a counting writer. serialize_into_slice serializes
once into caller-owned memory and returns the exact initialized length. If the
slice is too small, Error::BufferTooSmall contains the exact required size.
Slice deserialization supports nested borrowed strings and byte slices. Their
lifetime is tied to the input and no object or payload copy is performed.
Packed ASCII7 strings necessarily expand into owned text; raw adaptive UTF-8
can be returned as Cow::Borrowed.
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize)]
struct View<'a> {
name: &'a str,
#[serde(borrow)]
payload: &'a [u8],
}
let source = View { name: "edge", payload: b"frame" };
let config = rustbinary::options().with_limit(4096);
let mut storage = vec![0; config.serialized_size(&source)? as usize];
let written = config.serialize_into_slice(&mut storage, &source)?;
let view: View<'_> = config.deserialize(&storage[..written])?;
assert_eq!(view.payload, b"frame");
# Ok::<(), rustbinary::Error>(())The codec itself does not allocate on this path. A user-defined Serde implementation can still allocate internally.
Codec-owned allocation-free paths include serialized_size,
serialize_into_slice, adaptive encode_*_into_slice,
decode_i64_slice_into, decode_string_into_slice, and bit-packed caller
buffers. Reader-based decoding requires DeserializeOwned; returning a
reference into a temporary reader buffer would be unsound.
Borrowed parsing is true zero-copy for borrowed string and byte payloads. It is the Serde stream path; zero_copy.rs contains its pointer-range assertions. Use the separate archive surface for mapped object graphs.
The optional archive feature is a distinct storage format based on rkyv's
validated relative-pointer layout. build creates a 64-byte RustBinary
envelope followed by a little-endian archive with 32-bit relative pointers.
The envelope records a format version, fixed format flags, a non-zero
application schema ID, and checked payload/file lengths. rkyv is pinned because
an incompatible archive-layout dependency update requires a RustBinary format
version review.
MappedArchive::open enforces the file-size limit, validates the envelope,
schema, alignment, and complete relative-pointer graph once, then root()
performs no allocation or deserialization. Opening is unsafe: every process
must keep the mapped file immutable and untruncated for the mapping lifetime.
Publish a new file and atomically switch application references; never update a
mapped file in place. The schema ID is application-owned and must change after
an incompatible root layout change. It is an identity check, not cryptographic
authentication.
The complete mmap_archive.rs program writes a new file, drops the construction buffer, maps it read-only, validates nested data, and proves that strings, vectors, and child records point inside the mapping.
with_adaptive_encoding() retains the compact Serde profile and adds explicit
data-aware APIs. Strategy selection compares complete encoded sizes, uses a
stable tag in the frame, and validates canonical varints, padding, lengths,
delta overflow, and RLE runs while decoding.
let adaptive = rustbinary::options()
.with_limit(1 << 20)
.with_adaptive_encoding();
let values = [1000, 1001, 1002, 1003];
let required = adaptive.encoded_i64_slice_size(&values)?;
let mut output = vec![0; required];
adaptive.encode_i64_slice_into_slice(&mut output, &values)?;
assert_eq!(adaptive.decode_i64_vec(&output)?, values);
let mut decoded_values = [0_i64; 4];
adaptive.decode_i64_slice_into(&mut decoded_values, &output)?;
assert_eq!(decoded_values, values);
let encoded = adaptive.encode_string("telemetry/primary")?;
assert_eq!(adaptive.decode_string(&encoded)?, "telemetry/primary");
let mut decoded_text = [0_u8; 32];
assert_eq!(
adaptive.decode_string_into_slice(&mut decoded_text, &encoded)?,
"telemetry/primary"
);
# Ok::<(), rustbinary::Error>(())Adaptive frames are explicit because silently changing the representation of ordinary Serde fields would break protocol compatibility.
String frames contain strategy:u8, a canonical decoded-byte-length varint,
and payload. Strategy 0 is raw UTF-8; strategy 1 is ASCII7 packed
least-significant bit first. ASCII7 is eligible only when every byte is ASCII
and the complete packed representation is strictly smaller. Equal sizes select
raw UTF-8.
Integer collection frames compare Raw independent ZigZag values, Delta
with checked i128 reconstruction, and RunLength value/run pairs. Delta wins
only when strictly smaller than raw and no larger than RLE. RLE wins only when
strictly smaller than raw. Otherwise raw is canonical. Decoders validate
varints, counts, run lengths, delta overflow, padding, limits, and trailing
policy.
See adaptive_zero_alloc.rs for strategy inspection, borrowing, exact caller buffers, and insufficient-capacity errors.
With simd, simd_backend() selects AVX2, SSE2, NEON, or scalar code at
runtime. Adaptive ASCII classification and one-byte varint runs use these
kernels. All unaligned loads are bounds-checked by the safe dispatcher; unsafe
code is isolated in target-specific modules and unsafe_op_in_unsafe_fn is
denied crate-wide.
AVX-512BW, SVE, and SME are detected and reported separately. They are not selected today: wider vectors are not automatically faster for small codec records, SME is a matrix facility rather than a byte-scanning facility, and those backends require target hardware CI before becoming a wire-engine claim.
use rustbinary::{Fingerprint as _, Reflect as _, StaticSize as _};
#[derive(
serde::Serialize,
serde::Deserialize,
rustbinary::Fingerprint,
rustbinary::Reflect,
rustbinary::StaticSize,
)]
struct Header {
enabled: bool,
count: u16,
coordinates: [i32; 2],
}
let config = rustbinary::options().with_fingerprint();
let value = Header { enabled: true, count: 7, coordinates: [2, 3] };
let frame = config.serialize(&value)?;
let _: Header = config.deserialize(&frame)?;
assert!(Header::MAX_SIZE >= frame.len() - 16);
# Ok::<(), rustbinary::Error>(())Fingerprint hashes field and variant names, declared types, declaration
order, integer encoding, effective endianness, trailing policy, resource
limits, format, and CBOR deterministic mode. Native-endian fingerprints are
different on little- and big-endian targets.
The current FNV-1a-based fingerprint is a compatibility identifier, not a cryptographic hash. It must not replace AEAD, signatures, or authorization.
BitPacked accepts #[bits = N], rejects values that do not fit, uses the
minimum enum tag width, and validates unused padding bits. StaticSize excludes
dynamically sized collections by design.
Reflect emits allocation-free descriptors for type names, field names,
declared type tokens, declaration indexes, enum variants, and variant indexes.
It is runtime-readable metadata generated at compile time and requires no
registry. See metadata.rs.
The derive package has a dedicated
English guide
and Chinese guide.
They document the generated contracts, accepted data shapes, generic bounds,
#[bits = N] validation, compile-fail cases, and production integration
patterns.
The pipeline is explicit and ordered: serialize, optionally compress, then encrypt. Deterministic CBOR recursively sorts canonical map keys. Compression has a size threshold and only stores Zstandard output when it is smaller. Encryption authenticates both ciphertext and frame metadata and always uses a fresh nonce, so encrypted bytes are intentionally nondeterministic.
let secure = rustbinary::options()
.with_limit(16 * 1024 * 1024)
.with_cbor_format()
.with_deterministic_encoding()
.with_zstd_compression(3)
.with_compression_threshold(256)
.with_encryption(rustbinary::EncryptionKey::new([0xA5; 32]));
# let value = vec![1u32, 2, 3];
let frame = secure.serialize(&value)?;
assert_eq!(secure.deserialize::<Vec<u32>>(&frame)?, value);
# Ok::<(), rustbinary::Error>(())Applications must obtain keys from a real key-management system. Hard-coded keys are suitable only for tests.
Compression headers record raw and stored lengths. Decoders reject unknown flags, inconsistent length relationships, decompression-length mismatches, truncation, and configured-limit violations. Compression is retained only when its output is strictly smaller.
Encryption obtains a fresh 192-bit nonce from the operating system. The full
header is AEAD associated data, authenticating the algorithm ID, nonce, and
lengths with the ciphertext. EncryptionKey owns 32 bytes, redacts Debug,
and zeroizes on drop; key derivation, rotation, storage, and access control are
application/KMS responsibilities. See
secure_pipeline.rs.
The schema-evolution feature uses a stable schema ID, version, canonical
field-ID ordering, length-delimited fields, unknown-field skipping, defaults,
borrowed fields, and application-controlled migrations. Field IDs and schema
IDs remain explicit protocol decisions rather than hashes that can silently
change during refactoring.
The frame contains magic RBE1, format version, flags, stable schema ID,
schema version, field count, and length-delimited (field_id, payload) entries.
Encoders sort IDs and reject duplicates. Decoders require strictly increasing
IDs and validate all length arithmetic before slicing.
Application protocol rules:
- Assign one permanent schema ID to a compatible type family.
- Never reuse a field ID for a different meaning or incompatible type.
- Preserve the ID when renaming a Rust field.
- Add optional/defaulted fields for backward compatibility.
- Use the encoded version for deliberate semantic migrations.
- Inspect unknown fields when forwarding or preservation is required.
See schema_evolution.rs for complete V1/V2 upgrade and downgrade behavior with a rename, default, and borrowed field.
with_parallel_serialization() encodes independent batch elements on scoped
workers and emits an ordered length table, so scheduling does not affect output
bytes. It is intended for large independent records; small values should use
the normal single-value API.
serialize_into writes directly to std::io::Write. deserialize_from reads
owned values from std::io::Read. Slice decoding is the only API that can
return borrowed values. Every untrusted boundary should set both a byte limit
and a collection limit.
Parallel encoding is intended for independent, sufficiently large records.
Each item is encoded separately; an ordered u64 length table and source-order
payload section make output independent of worker scheduling. Small values may
be slower because worker and merge overhead can dominate. See
parallel_batch.rs.
Compression and encryption stream readers consume one declared frame when
passed &mut R, leaving later frames unread. Header relationships and
configured raw/plaintext limits are checked before body allocation.
- Endian, integer mode, and enum representation are explicit.
- Adaptive tags and tie-breaking are canonical.
- Bit-packed terminal padding must be zero.
- Schema-evolution fields are sorted by stable numeric ID.
- Parallel batches retain source order.
- Deterministic CBOR recursively sorts canonical map keys.
- Floating-point IEEE bit patterns, including NaN payloads, are preserved.
Ordinary HashMap iteration is randomized and is not deterministic. Use
BTreeMap, another ordered serializer, or deterministic CBOR. Encrypted frames
are intentionally nondeterministic because nonce reuse would be a security
failure.
- Boolean and option tags are one-byte
0or1values. - Floats preserve their IEEE 754 bit pattern; endianness is explicit.
- Variable integers reject marker 255 and non-minimal encodings.
- Struct fields are encoded in declaration order; names are not in binary payloads.
- Ordinary maps preserve Serde iteration order and are not deterministic.
- Deterministic map serialization requires deterministic CBOR or an ordered map.
- Compression and encryption frames validate versions, flags, lengths, and limits.
- Stream decoders validate frame length relationships and configured limits before body allocation.
- Decryption authenticates before deserialization.
- Fingerprints are compatibility checks, not cryptographic authentication.
- User-defined Serde implementations may allocate or reject borrowed visitors.
At every untrusted boundary, set realistic byte and collection limits, reject trailing bytes unless an outer protocol owns them, authenticate adversarial data, and treat decompression/deserialization errors as input failures.
All operations return rustbinary::Result<T>. Error preserves I/O errors and
has structured variants for limits, capacity, frames, schemas, compression,
encryption, bit packing, adaptive data, worker failure, and malformed primitive
values. It is #[non_exhaustive]; downstream exhaustive matches need a fallback
arm. Frame offsets, length sums, delta reconstruction, and integer narrowing
are checked instead of relying on panic recovery.
Error::category() provides the stable operational mapping to UserInput,
Protocol, Configuration, or InternalBug.
cargo fmt --all -- --check
cargo test --workspace --all-targets --all-features
cargo test --workspace --all-features --release
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo doc --workspace --all-features --no-deps
cargo bench --bench codec_comparison
The benchmark compares RustBinary's owned and caller-buffer Compact V1 paths over the same Serde shapes. It validates each round trip before collecting nine calibrated samples and prints the median as raw Markdown.
| Example | Scope | Command |
|---|---|---|
| complete.rs | End-to-end all-feature composition | cargo run --example complete --all-features |
| core_codec.rs | Bounded Core, caller buffers, borrowing, trailing and error policy | cargo run --example core_codec |
| zero_copy.rs | Nested borrowing and pointer proof | cargo run --example zero_copy |
| mmap_archive.rs | Validated read-only mmap object graph and pointer proof | cargo run --example mmap_archive --features archive |
| adaptive_zero_alloc.rs | Adaptive decisions and caller buffers | cargo run --example adaptive_zero_alloc --features adaptive |
| secure_pipeline.rs | Deterministic CBOR, compression, AEAD | cargo run --example secure_pipeline --features cbor,compression,encryption |
| schema_evolution.rs | Bidirectional schema V1/V2 | cargo run --example schema_evolution --features schema-evolution |
| parallel_batch.rs | Ordered multi-worker batches | cargo run --example parallel_batch --features parallel |
| metadata.rs | Fingerprint, reflection, bounds, packing | cargo run --example metadata --features bit-packing,derive,fingerprint,reflection,static-size |
Package metadata builds docs.rs with all features. Public modules are grouped by subsystem and feature-gated APIs receive automatic docs.rs labels. Strict local documentation validation on PowerShell:
$env:RUSTDOCFLAGS='-D warnings'
cargo doc --workspace --all-features --no-depsVersioned wrappers reject unknown versions and reserved flags instead of guessing. Before 1.0, wire changes may occur between minor releases and must be called out in release notes. Long-lived deployments should pin the version, record the complete configuration, keep golden vectors, and use explicit schema IDs.
- Casting arbitrary Rust structs directly from serialized memory
- Mutable shared-memory object graphs or in-place updates to mapped files
- Wrapping blocking I/O in a misleading async facade
- Automatically sorting randomized maps in the core profile
- Claiming AVX-512/SVE acceleration without tested kernels
- Replacing application key management, authorization, or schema governance
RustBinary is licensed under the Apache License, Version 2.0.
You may use, reproduce, modify, and redistribute the project under the terms of that license. Redistributions must preserve the copyright notice, license text, and required attribution notices. Changes to the source should be identified clearly, and the Apache License patent terms and disclaimer apply.
The complete legal text is in LICENSE. This project is provided
without warranties or conditions of any kind.