Skip to content

About

Header-only C++ Protobuf decoders generated from your schema — arena tree or zero-copy streaming, decode-only, fully validating, measurably faster than protoc and protozero.

Topics

Resources

Contributing

Security policy

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

RapidProto

CI Rust CI Docs Release crates.io docs.rs License: Apache-2.0

RapidProto generates Protobuf decoders for C++ and Rust from your .proto schemas. On every payload in the benchmarks they decode faster than protoc's generated C++ code and prost: 1.2 to 7 times as fast in C++, and 2.6 to 12 times in Rust.

On Dataset, a benchmark payload of 2000 mixed records, the C++ arena decoder has about 7 times the throughput of protoc's generated C++ code allocating from a google::protobuf::Arena. The Rust arena decoder has about 12 times the throughput of prost. The optimizations article shows where the speed comes from, one optimization at a time.

Schemas can be proto2, proto3, or editions 2023 and 2024, with groups, maps and oneofs. The generator parses them itself and carries its own copy of the well-known types, so you don't need protoc.

On malformed input, a decode fails with an error instead of crashing. On every pull request, the C++ wire reader and both C++ decoders are fuzzed under ASan and UBSan, and the Rust arena decoders under ASan. SECURITY.md has the details. For untrusted input, also limit how much memory the arena may take with Arena::set_capacity_limit. The Rust crate documents it under untrusted input.

Two decoders

Each schema can have two decoders. The default is the arena decoder. It decodes the whole message into an arena you own and gives you an accessor for each field.

The streaming decoder is generated when you ask for it. It reads the input once and calls your handler for each field you selected, as the fields arrive. No message is built. Each quick start below ends with a short streaming example.

Arena decoder Streaming decoder
Allocation One arena, which you own None
Reading a field again Any field, in any order, as often as you like Only by decoding the input again
Memory The whole decoded message Whatever your handlers store
Best for Code that works with the message as an object A few fields from a large message or a stream

One program can use both decoders of a schema. A streaming handler that receives a sub-message can also hand its bytes to the arena decoder. In C++ it gets them from rp_bytes(), which docs/using-both-models.md walks through. In Rust it calls bytes().

When not to use RapidProto

RapidProto can't build or change messages. It has no encoder and decoded messages are read-only, so a program that does either needs another Protobuf library for that part. There's no reflection either.

RapidProto doesn't check strings for valid UTF-8 while decoding, where prost and protoc's proto3 code do. Skipping the check accounts for part of the speed difference. The benchmark page measures its cost for Rust. In Rust you can check one string when you need to, with ProtoStr::to_str. In C++ the check is yours to do.

A few Protobuf behaviors differ. Every enum is open: a value the schema doesn't list comes through as its number, even in an enum declared closed. Extensions aren't decoded; they arrive as unknown fields. A proto2 [default = X] isn't applied. A singular sub-message that appears twice fails the arena decode instead of being merged. gRPC is out of scope too: service and rpc definitions are skipped.

Decoded strings and bytes point into the input, which has to stay alive while you use them. With the arena decoder, so does the arena. If that's awkward, rapidproto::decode_owned<T> in C++ and Msg::decode_owned in Rust bundle the input and the arena with the message.

Expect breaking changes, too. The C++ side hasn't reached 1.0, and the Rust crates are test releases (Compatibility).

Quick start

syntax = "proto3";
package example;

message Person {
  string name = 1;
  uint32 id = 2;
  repeated string email = 3;
  Address address = 4;
}

message Address {
  string city = 1;
  string country = 2;
}

Save the schema as person.proto and encode a test message with protoc. Neither quick start needs protoc after this step.

echo 'name: "Ada" id: 7 address { city: "London" }' |
    protoc --encode=example.Person -I. person.proto > person.bin

C++

You need a C++17 compiler and the rapidprotoc generator. The latest release has unsigned binaries for Linux on x86-64 and arm64, macOS on Apple silicon and Windows on x86-64. If Gatekeeper blocks rapidprotoc, clear its quarantine flag with xattr -d com.apple.quarantine rapidprotoc; on Windows, click past the SmartScreen prompt. On an Intel Mac, or to build the generator yourself on Linux or macOS, use CMake 3.21 or newer:

cmake --preset release
cmake --build --preset release --target rapidprotoc   # writes build/release/rapidprotoc

Put rapidprotoc on your PATH, then generate the arena decoder:

rapidprotoc -I. --out-dir=out person.proto
# writes out/person.rp.hpp, out/person.rp.common.hpp and out/rapidproto/*.hpp

person.rp.hpp is the arena decoder and person.rp.common.hpp holds the schema's enums. The headers in out/rapidproto/ are a copy of the runtime. It's all headers on top of the standard library, with nothing to link.

// read_person.cpp
#include "person.rp.hpp"

#include <fstream>
#include <iostream>
#include <sstream>

namespace ex = rp::arena::example;

int main() {
    std::ostringstream file;
    file << std::ifstream("person.bin", std::ios::binary).rdbuf();
    const std::string input = file.str();

    rapidproto::Arena arena;
    rapidproto::ArenaDecodeError err;
    const ex::Person* person = ex::Person::decode(rapidproto::ByteView(input), arena, &err);
    if (person == nullptr) {
        return 1;  // malformed input: err.code and err.offset say why
    }

    std::uint32_t id = person->id();
    std::string_view name = person->name();  // points into input
    if (const ex::Address* address = person->address()) {  // nullptr when absent
        std::cout << name << " (" << id << ") lives in " << address->city() << '\n';
    }
}

Each package gets a namespace under rp::arena::. If your input is a const std::uint8_t* and a size, rapidproto::byte_view(ptr, size) makes the rapidproto::ByteView for you. Repeated fields, maps, oneofs and optional fields return different types.

g++ -std=c++17 -Iout read_person.cpp -o read_person

From the directory with person.bin, ./read_person prints Ada (7) lives in London.

In a CMake project, rapidproto_generate() runs the generator at build time. The same page covers FetchContent, find_package, and the Conan recipe and vcpkg overlay port in this repository.

For the streaming decoder, pass --stream and the generator writes person.rp.stream.hpp instead. --arena --stream writes both. Generate both, add #include "person.rp.stream.hpp" to read_person.cpp, and replace everything in main after input is read with this, which prints the name and the id:

namespace ex = rp::stream::example;
rapidproto::DecodeStatus status = ex::Person{rapidproto::ByteView(input)}.decode(
    [](ex::Person::name, std::string_view name) { std::cout << name << '\n'; },
    [](ex::Person::id, std::uint32_t id) { std::cout << id << '\n'; });
if (!status.ok()) {
    return 1;  // malformed input
}

Each handler's first parameter type names the field it wants.

Rust

You need Rust 1.91 or newer. rapidproto is the no_std runtime that the generated code uses. rapidproto-build generates that code from your build script. Pin both to the same version:

# Cargo.toml
[dependencies]
rapidproto = "=0.1.0-alpha.2"

[build-dependencies]
rapidproto-build = "=0.1.0-alpha.2"

Generated code works only with the runtime from the same release. When the two differ, the build usually fails with an error asking you to match them.

Put person.proto and person.bin next to Cargo.toml. Config resolves schema paths against the package root, where build scripts run.

// build.rs
fn main() {
    rapidproto_build::Config::new("crate::schema")
        .compile(&["person.proto"])
        .expect("generate decoders");
}

compile names its output after the first schema, here person.rs in OUT_DIR. rapidproto-build compiles the generator with your C++ compiler. Without one it warns and falls back to a slower WebAssembly build. The crate's notes on how the generator runs show the profile setting that makes the WebAssembly generator fast.

Then decode the test message:

// src/main.rs
mod schema {
    include!(concat!(env!("OUT_DIR"), "/person.rs"));
}

use schema::arena::example::Person;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = std::fs::read("person.bin")?;

    let arena = rapidproto::Arena::new();
    let person = Person::decode(&input, &arena)?;

    let id = person.id();
    let name = person.name().to_str()?; // borrows from input, checks UTF-8
    if let Some(address) = person.address() {
        println!("{name} ({id}) lives in {}", address.city());
    }
    Ok(())
}

Config::new takes the path of the module that holds the include!, here crate::schema. Arena types sit under <module path>::arena::<package>, as in C++. The example module on docs.rs shows what each kind of field returns. cargo run from the package root prints Ada (7) lives in London.

Streaming in Rust needs .stream(true) on the Config in build.rs. In src/main.rs, swap the arena import for the use line below. Then, in main, replace the lines between reading input and Ok(()) with the decode call:

use schema::stream::example::{person_stream, Person};

Person::new(&input)?.decode(person_stream! {
    name => |name| println!("{name}"),
    id => |id| println!("{id}"),
})?;

The generated person_stream! macro pairs each field name with its handler. The stream module docs explain the rest.

Documentation

The manual is written for C++ and is also on the project site. Only its benchmarks page covers Rust.

The Rust guide is on docs.rs. rapidproto documents decoding, including Debug output, opt-in serde::Serialize (there's no Deserialize), and Arena::with_buffer, which lets a decode run without an allocator once the default alloc feature is off. rapidproto-build documents generating. That includes the rapidproto-rust command, for projects that commit the generated code instead of generating it in build.rs.

examples/consumer and rust/examples/consumer each decode one schema with both decoders. The C++ one builds with rapidproto_generate() and adds a second build with a decode profile. For something larger, examples/osm-pbf and rust/examples/osm-pbf read OpenStreetMap files.

Compatibility

C++ and Rust have separate releases. The badges at the top show the current version of each. Before 1.0, a C++ minor release may break things but a patch release won't. CHANGELOG.md lists the changes in every C++ release and opens with the full compatibility policy, including what changes at 1.0.

CI builds and tests the C++ side on Linux, macOS and Windows with GCC, Clang, AppleClang and MSVC. The C++ arena decoder stops at 100 levels of nesting, a limit sized for the 8 MB stack of Linux and macOS. Windows gives a program 1 MB, so link programs that decode untrusted input there with /STACK:8388608.

The Rust crates are test releases. Until they're stable, any release may change their API and the code the generator writes. CI runs the full Rust test suite on Linux. On macOS and Windows it builds the generator and runs it in a smoke test.

Contributing and security

  • CONTRIBUTING.md: building RapidProto, running the gate (./check.sh) and updating the golden files
  • architecture.md: the internals and the invariants a change must keep
  • rust/README.md: the Cargo workspace and the Rust benchmarks
  • SECURITY.md: the threat model and which releases get fixes

Please report vulnerabilities through GitHub's private vulnerability reporting, not in a public issue.

License

RapidProto is licensed under Apache-2.0, and so are the runtime headers rapidprotoc copies into your output and both Rust crates. RapidProto claims no rights in the code generated from your schemas.

The generator includes Google's well-known type .proto files (copyright 2008 Google Inc., BSD-3-Clause), and the wasm generator in rapidproto-build bundles wasi-libc and the LLVM C++ runtime under their own licenses. NOTICE and THIRD_PARTY_NOTICES.md list them. Test and benchmark dependencies such as Catch2, protozero and prost aren't distributed with RapidProto.

About

Header-only C++ Protobuf decoders generated from your schema — arena tree or zero-copy streaming, decode-only, fully validating, measurably faster than protoc and protozero.

Topics

Resources

Contributing

Security policy

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages