feat!: Make schema parser reuse the JSON allocations - #656
Merged
Conversation
This changes the schema parser from iterating over the JSON into consuming it. This allows reusing the existing String allocations (and also collection allocations if the sizes match and the compiler can make it work). Because we remove everything from the JSON while consuming it, custom attributes is just everything leftover. Therefore this also fixes #654 I've tried my best to improve the error messages where possible. This is a breaking change because the input for `Schema::parse` changes from a reference to an owned `Value`. Users can fix their usage by calling `.clone()` where needed.
Kriskras99
force-pushed
the
feat/improve_schema_parser
branch
from
September 4, 2026 19:42
fbd9c72 to
f6d4c0b
Compare
martin-g
approved these changes
Sep 7, 2026
|
Hey @Kriskras99 , I have pulled down your PR and can confirm my Avro test passes and so does my Iceberg test! Thank you very much for getting this done. If it helps the tests I have are these. They failed before, and pass now. Feel free to add them to your PR (or a followup) if you find them helpful. I had them in: #[test]
fn it_should_preserve_map_logical_type_on_outer_item() -> TestResult {
let raw_schema = r#"{
"type": "array",
"logicalType": "map",
"items": {
"type": "record",
"name": "k12_v13",
"fields": [
{
"name": "key",
"type": "int",
"field-id": 12
},
{
"name": "value",
"type": "string",
"field-id": 13
}
]
}
}"#;
let schema = Schema::parse_str(raw_schema)?;
let output = serde_json::to_string_pretty(&schema).unwrap();
pretty_assertions::assert_eq!(
r#"{
"type": "array",
"items": {
"type": "record",
"name": "k12_v13",
"fields": [
{
"name": "key",
"type": "int",
"field-id": 12
},
{
"name": "value",
"type": "string",
"field-id": 13
}
]
},
"logicalType": "map"
}"#,
output
);
let logical_type = schema.custom_attributes().unwrap().get("logicalType");
assert_eq!(
logical_type,
Some(&serde_json::Value::String("map".to_string()))
);
Ok(())
}
#[test]
fn it_should_preserve_map_logical_type_on_inner_item() -> TestResult {
let raw_schema = r#"{
"type": "record",
"name": "test_record",
"fields": [
{
"name": "example_map",
"type": {
"type": "array",
"logicalType": "map",
"items": {
"type": "record",
"name": "k12_v13",
"fields": [
{
"name": "key",
"type": "int",
"field-id": 12
},
{
"name": "value",
"type": "string",
"field-id": 13
}
]
}
}
}
]
}"#;
let schema = Schema::parse_str(raw_schema)?;
let Schema::Record(record) = &schema else {
panic!("Expected a record schema");
};
let example_map_schema = &record.fields[0].schema;
let logical_type = example_map_schema
.custom_attributes()
.unwrap()
.get("logicalType");
assert_eq!(
logical_type,
Some(&serde_json::Value::String("map".to_string()))
);
Ok(())
} |
Contributor
Author
|
Perfect! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changes the schema parser from iterating over the JSON into consuming it. This allows reusing the existing String allocations (and also collection allocations if the sizes match and the compiler can make it work).
Because we remove everything from the JSON while consuming it, custom attributes is just everything leftover. Therefore this also fixes #654
I've tried my best to improve the error messages where possible.
This is a breaking change because the input for
Schema::parsechangesfrom a reference to an owned
Value. Users can fix their usage bycalling
.clone()where needed.@JosephLenton I've added your schema to one of the tests, can you verify everything works in your application?