Skip to content

feat!: Make schema parser reuse the JSON allocations - #656

Merged
Kriskras99 merged 5 commits into
mainfrom
feat/improve_schema_parser
Sep 7, 2026
Merged

feat!: Make schema parser reuse the JSON allocations#656
Kriskras99 merged 5 commits into
mainfrom
feat/improve_schema_parser

Conversation

@Kriskras99

Copy link
Copy Markdown
Contributor

This changes the schema parser from iterating over the JSON into consuming it. This allows reusing the existing String allocations (and also collection allocations if the sizes match and the compiler can make it work).
Because we remove everything from the JSON while consuming it, custom attributes is just everything leftover. Therefore this also fixes #654

I've tried my best to improve the error messages where possible.

This is a breaking change because the input for Schema::parse changes
from a reference to an owned Value. Users can fix their usage by
calling .clone() where needed.

@JosephLenton I've added your schema to one of the tests, can you verify everything works in your application?

This changes the schema parser from iterating over the JSON into consuming it.
This allows reusing the existing String allocations (and also collection
allocations if the sizes match and the compiler can make it work).

Because we remove everything from the JSON while consuming it, custom attributes
is just everything leftover. Therefore this also fixes #654

I've tried my best to improve the error messages where possible.

This is a breaking change because the input for `Schema::parse` changes
from a reference to an owned `Value`. Users can fix their usage by
calling `.clone()` where needed.
@Kriskras99
Kriskras99 force-pushed the feat/improve_schema_parser branch from fbd9c72 to f6d4c0b Compare September 4, 2026 19:42
@martin-g martin-g added this to the 0.23.0 milestone Sep 5, 2026
Comment thread avro/src/schema/mod.rs Outdated
Comment thread avro/src/schema/mod.rs Outdated
Comment thread avro/src/schema/parser.rs Outdated
@JosephLenton

Copy link
Copy Markdown

Hey @Kriskras99 , I have pulled down your PR and can confirm my Avro test passes and so does my Iceberg test! Thank you very much for getting this done.

If it helps the tests I have are these. They failed before, and pass now. Feel free to add them to your PR (or a followup) if you find them helpful.

I had them in: avro/tests/schema.rs

#[test]
fn it_should_preserve_map_logical_type_on_outer_item() -> TestResult {
    let raw_schema = r#"{
        "type": "array",
        "logicalType": "map",
        "items": {
            "type": "record",
            "name": "k12_v13",
            "fields": [
                {
                    "name": "key",
                    "type": "int",
                    "field-id": 12
                },
                {
                    "name": "value",
                    "type": "string",
                    "field-id": 13
                }
            ]
        }
    }"#;

    let schema = Schema::parse_str(raw_schema)?;

    let output = serde_json::to_string_pretty(&schema).unwrap();
    pretty_assertions::assert_eq!(
        r#"{
  "type": "array",
  "items": {
    "type": "record",
    "name": "k12_v13",
    "fields": [
      {
        "name": "key",
        "type": "int",
        "field-id": 12
      },
      {
        "name": "value",
        "type": "string",
        "field-id": 13
      }
    ]
  },
  "logicalType": "map"
}"#,
        output
    );

    let logical_type = schema.custom_attributes().unwrap().get("logicalType");
    assert_eq!(
        logical_type,
        Some(&serde_json::Value::String("map".to_string()))
    );

    Ok(())
}

#[test]
fn it_should_preserve_map_logical_type_on_inner_item() -> TestResult {
    let raw_schema = r#"{
        "type": "record",
        "name": "test_record",
        "fields": [
            {
            "name": "example_map",
            "type": {
                "type": "array",
                "logicalType": "map",
                "items": {
                    "type": "record",
                    "name": "k12_v13",
                    "fields": [
                        {
                            "name": "key",
                            "type": "int",
                            "field-id": 12
                        },
                        {
                            "name": "value",
                            "type": "string",
                            "field-id": 13
                        }
                    ]
                }
            }
            }
        ]
    }"#;

    let schema = Schema::parse_str(raw_schema)?;
    let Schema::Record(record) = &schema else {
        panic!("Expected a record schema");
    };
    let example_map_schema = &record.fields[0].schema;
    let logical_type = example_map_schema
        .custom_attributes()
        .unwrap()
        .get("logicalType");
    assert_eq!(
        logical_type,
        Some(&serde_json::Value::String("map".to_string()))
    );

    Ok(())
}

@Kriskras99

Copy link
Copy Markdown
Contributor Author

Perfect!

@Kriskras99
Kriskras99 merged commit f48baf4 into main Sep 7, 2026
16 checks passed
@Kriskras99
Kriskras99 deleted the feat/improve_schema_parser branch September 7, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

regression bug, logicalType is dropped from custom attributes

3 participants