-
Notifications
You must be signed in to change notification settings - Fork 101
chore: simplify grpc file descriptor codegen #1718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Mirko-von-Leipzig
wants to merge
1
commit into
next
Choose a base branch
from
mirko/build-codegen
base: next
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,79 +1,105 @@ | ||
| use std::ffi::OsStr; | ||
| use std::path::PathBuf; | ||
|
|
||
| use fs_err as fs; | ||
| use miette::{Context, IntoDiagnostic}; | ||
| use miette::{IntoDiagnostic, miette}; | ||
| use protox::prost::Message; | ||
|
|
||
| const RPC_PROTO: &str = "rpc.proto"; | ||
| // Unified internal store API (store.Rpc, store.BlockProducer, store.NtxBuilder). | ||
| // We compile the same file three times to preserve existing descriptor names. | ||
| const STORE_RPC_PROTO: &str = "internal/store.proto"; | ||
| const STORE_NTX_BUILDER_PROTO: &str = "internal/store.proto"; | ||
| const STORE_BLOCK_PRODUCER_PROTO: &str = "internal/store.proto"; | ||
| const BLOCK_PRODUCER_PROTO: &str = "internal/block_producer.proto"; | ||
| const REMOTE_PROVER_PROTO: &str = "remote_prover.proto"; | ||
| const VALIDATOR_PROTO: &str = "internal/validator.proto"; | ||
|
|
||
| const RPC_DESCRIPTOR: &str = "rpc_file_descriptor.bin"; | ||
| const STORE_RPC_DESCRIPTOR: &str = "store_rpc_file_descriptor.bin"; | ||
| const STORE_NTX_BUILDER_DESCRIPTOR: &str = "store_ntx_builder_file_descriptor.bin"; | ||
| const STORE_BLOCK_PRODUCER_DESCRIPTOR: &str = "store_block_producer_file_descriptor.bin"; | ||
| const BLOCK_PRODUCER_DESCRIPTOR: &str = "block_producer_file_descriptor.bin"; | ||
| const REMOTE_PROVER_DESCRIPTOR: &str = "remote_prover_file_descriptor.bin"; | ||
| const VALIDATOR_DESCRIPTOR: &str = "validator_file_descriptor.bin"; | ||
|
|
||
| /// Generates Rust protobuf bindings from .proto files. | ||
| /// Compiles each gRPC service definitions into a | ||
| /// [`FileDescriptorSet`](tonic_prost_build::FileDescriptorSet) and exposes it as a function: | ||
| /// | ||
| /// This is done only if `BUILD_PROTO` environment variable is set to `1` to avoid running the | ||
| /// script on crates.io where repo-level .proto files are not available. | ||
| /// ```rust | ||
| /// fn <service>_api_descriptor() -> FileDescriptorSet; | ||
| /// ``` | ||
| fn main() -> miette::Result<()> { | ||
| build_rs::output::rerun_if_changed("./proto"); | ||
| build_rs::output::rerun_if_changed("Cargo.toml"); | ||
|
|
||
| let out_dir = build_rs::input::out_dir(); | ||
| let crate_root = build_rs::input::cargo_manifest_dir(); | ||
| let proto_src_dir = crate_root.join("proto"); | ||
| let includes = &[proto_src_dir]; | ||
|
|
||
| let rpc_file_descriptor = protox::compile([RPC_PROTO], includes)?; | ||
| let rpc_path = out_dir.join(RPC_DESCRIPTOR); | ||
| fs::write(&rpc_path, rpc_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing rpc file descriptor")?; | ||
|
|
||
| let remote_prover_file_descriptor = protox::compile([REMOTE_PROVER_PROTO], includes)?; | ||
| let remote_prover_path = out_dir.join(REMOTE_PROVER_DESCRIPTOR); | ||
| fs::write(&remote_prover_path, remote_prover_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing remote prover file descriptor")?; | ||
|
|
||
| let store_rpc_file_descriptor = protox::compile([STORE_RPC_PROTO], includes)?; | ||
| let store_rpc_path = out_dir.join(STORE_RPC_DESCRIPTOR); | ||
| fs::write(&store_rpc_path, store_rpc_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing store rpc file descriptor")?; | ||
|
|
||
| let store_ntx_builder_file_descriptor = protox::compile([STORE_NTX_BUILDER_PROTO], includes)?; | ||
| let store_ntx_builder_path = out_dir.join(STORE_NTX_BUILDER_DESCRIPTOR); | ||
| fs::write(&store_ntx_builder_path, store_ntx_builder_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing store ntx builder file descriptor")?; | ||
|
|
||
| let store_block_producer_file_descriptor = | ||
| protox::compile([STORE_BLOCK_PRODUCER_PROTO], includes)?; | ||
| let store_block_producer_path = out_dir.join(STORE_BLOCK_PRODUCER_DESCRIPTOR); | ||
| fs::write(&store_block_producer_path, store_block_producer_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing store block producer file descriptor")?; | ||
|
|
||
| let block_producer_file_descriptor = protox::compile([BLOCK_PRODUCER_PROTO], includes)?; | ||
| let block_producer_path = out_dir.join(BLOCK_PRODUCER_DESCRIPTOR); | ||
| fs::write(&block_producer_path, block_producer_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing block producer file descriptor")?; | ||
|
|
||
| let validator_file_descriptor = protox::compile([VALIDATOR_PROTO], includes)?; | ||
| let validator_path = out_dir.join(VALIDATOR_DESCRIPTOR); | ||
| fs::write(&validator_path, validator_file_descriptor.encode_to_vec()) | ||
| .into_diagnostic() | ||
| .wrap_err("writing validator file descriptor")?; | ||
| let schema_dir = build_rs::input::cargo_manifest_dir().join("proto"); | ||
|
|
||
| // Codegen which will hold the file descriptor functions. | ||
| // | ||
| // `protox::prost::Message` is a trait which brings into scope the encoding and decoding of file | ||
| // descriptors. This is required so because we serialize the descriptors in code as a `Vec<u8>` | ||
| // and then decode it again inline. | ||
| let mut code = codegen::Scope::new(); | ||
| code.import("tonic_prost_build", "FileDescriptorSet"); | ||
| code.import("protox::prost", "Message"); | ||
|
|
||
| // We split our gRPC services into public and internal. | ||
| // | ||
| // This is easy to do since public services are listed in the root of the schema folder, | ||
| // and internal services are nested in the `internal` folder. | ||
| for public_api in proto_files_in_directory(&schema_dir)? { | ||
| let file_descriptor_fn = generate_file_descriptor(&public_api, &schema_dir)?; | ||
| code.push_fn(file_descriptor_fn); | ||
| } | ||
|
|
||
| // Internal gRPC services need an additional feature gate `#[cfg(feature = "internal")]`. | ||
| for internal_api in proto_files_in_directory(&schema_dir.join("internal"))? { | ||
| let mut file_descriptor_fn = generate_file_descriptor(&internal_api, &schema_dir)?; | ||
| file_descriptor_fn.attr("cfg(feature = \"internal\")"); | ||
| code.push_fn(file_descriptor_fn); | ||
| } | ||
|
|
||
| fs::write(out_dir.join("file_descriptors.rs"), code.to_string()).into_diagnostic()?; | ||
|
|
||
| Ok(()) | ||
| } | ||
|
|
||
| /// The list of `*.proto` files in the given directory. | ||
| /// | ||
| /// Does _not_ recurse into folders; only top level files are returned. | ||
| fn proto_files_in_directory(directory: &PathBuf) -> Result<Vec<PathBuf>, miette::Error> { | ||
| let mut proto_files = Vec::new(); | ||
| for entry in fs::read_dir(directory).into_diagnostic()? { | ||
| let entry = entry.into_diagnostic()?; | ||
|
|
||
| // Skip non-files | ||
| if !entry.file_type().into_diagnostic()?.is_file() { | ||
| continue; | ||
| } | ||
|
|
||
| // Skip non-protobuf files | ||
| if PathBuf::from(entry.file_name()).extension().is_none_or(|ext| ext != "proto") { | ||
| continue; | ||
| } | ||
|
|
||
| proto_files.push(entry.path()); | ||
| } | ||
| Ok(proto_files) | ||
| } | ||
|
|
||
| /// Creates a function which emits the file descriptor of the given gRPC service file. | ||
| /// | ||
| /// The function looks as follows: | ||
| /// | ||
| /// ```rust | ||
| /// fn <file_stem>_api_descriptor() -> FileDescriptorSet { | ||
| /// FileDescriptorSet::decode(vec![<encoded>].as_slice()) | ||
| /// .expect("encoded file descriptor should decode") | ||
| /// } | ||
| /// ``` | ||
| /// | ||
| /// where `<encoded>` is bytes of the compiled gRPC service. | ||
| fn generate_file_descriptor( | ||
| grpc_service: &PathBuf, | ||
| includes: &PathBuf, | ||
| ) -> Result<codegen::Function, miette::Error> { | ||
| let file_name = grpc_service | ||
| .file_stem() | ||
| .and_then(OsStr::to_str) | ||
| .ok_or_else(|| miette!("invalid file name for {grpc_service:?}"))?; | ||
|
|
||
| let file_descriptor = protox::compile([grpc_service], includes)?; | ||
| let file_descriptor = file_descriptor.encode_to_vec(); | ||
|
|
||
| let mut f = codegen::Function::new(format!("{file_name}_api_descriptor")); | ||
| f.vis("pub") | ||
| .ret("FileDescriptorSet") | ||
| .line(format!("FileDescriptorSet::decode(vec!{file_descriptor:?}.as_slice())")) | ||
| .line(".expect(\"we just encoded this so it should decode\")"); | ||
|
|
||
| Ok(f) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,57 +1 @@ | ||
| use protox::prost::Message; | ||
| use tonic_prost_build::FileDescriptorSet; | ||
|
|
||
| /// Returns the Protobuf file descriptor for the RPC API. | ||
| pub fn rpc_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", "rpc_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
|
|
||
| /// Returns the Protobuf file descriptor for the remote prover API. | ||
| pub fn remote_prover_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", "remote_prover_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
|
|
||
| /// Returns the Protobuf file descriptor for the store RPC API. | ||
| #[cfg(feature = "internal")] | ||
| pub fn store_rpc_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", "store_rpc_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
|
|
||
| /// Returns the Protobuf file descriptor for the store NTX builder API. | ||
| #[cfg(feature = "internal")] | ||
| pub fn store_ntx_builder_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = | ||
| include_bytes!(concat!(env!("OUT_DIR"), "/", "store_ntx_builder_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
|
|
||
| /// Returns the Protobuf file descriptor for the store block producer API. | ||
| #[cfg(feature = "internal")] | ||
| pub fn store_block_producer_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = | ||
| include_bytes!(concat!(env!("OUT_DIR"), "/", "store_block_producer_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
|
|
||
| /// Returns the Protobuf file descriptor for the block-producer API. | ||
| #[cfg(feature = "internal")] | ||
| pub fn block_producer_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", "block_producer_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
|
|
||
| /// Returns the Protobuf file descriptor for the validator API. | ||
| pub fn validator_api_descriptor() -> FileDescriptorSet { | ||
| let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/", "validator_file_descriptor.bin")); | ||
| FileDescriptorSet::decode(&bytes[..]) | ||
| .expect("bytes should be a valid file descriptor created by build.rs") | ||
| } | ||
| include!(concat!(env!("OUT_DIR"), "/file_descriptors.rs")); | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
comment: it's unfortunate
build-rsdoesn't provide this as well