On-the-fly schema creation from json-ld to parquet

[eldondevat]

I am interested in processing some json-ld files on the fly through arrow-go to parquet files. I recently found a strategy from pyarrow to accomplish approximately what I'm hoping for: https://stackoverflow.com/a/67126122 . Is there a similarly trivial strategy with arrow-go, something that can infer a schema and output a parquet file on-the-fly? My primary interest here in a more compact data representation, and parquet files ended up being about 10% the size of my existing representation.

[zeroshade]

Currently, we have not implemented inferring a schema from JSON for arrow-go, if you can define some kind of schema beforehand and parse the JSON based on that schema then you can do it fairly easily in arrow-go. (creating record batches/table from JSON and writing a Parquet file from that table all exist).

[eldondevat]

I did some more research today, and found this: https://github.com/loicalleyne/bodkin , seems to be very similar to what I was looking for. Thank you!