Discuss if we should change serialization format

[e-lo]

This issue proposes revisiting whether HDF5 remains the best long-term container choice for OMX, or whether a modern alternative (or optional backend) would better support current and future use cases.

The goal is not to break OMX semantics, but to evaluate whether the container layer could evolve while preserving:

• Stable matrix semantics
• Efficient sparse and dense storage
• Long-term reproducibility

Background

OMX currently uses HDF5 as its underlying container, a design choice discussed early in the project’s history. At the time, HDF5 provided a mature, high-performance, cross-platform solution for large matrix storage.

Since then, the data ecosystem—especially in Python—has shifted significantly toward columnar, cross-language formats such as Apache Arrow and Parquet, which emphasize interoperability, cloud friendliness, and zero-copy data exchange.

Questions for discussion

1. Does HDF5 continue to meet OMX’s needs in modern Python and cloud-based workflows?
2. Are there known pain points with HDF5 (tooling, deployment, performance, maintenance)?
3. Could Arrow IPC / Feather or Parquet realistically serve as:
   a. A replacement container?
   b. An optional backend?
   ...with critical current OMX features(random access, slicing, determinism)

Prerequisites

Need to evolve our governance model to support a decision this significant.

Possible outcomes

1. Affirm HDF5 as the long-term container and document why
2. Support an alternative container backend while retaining OMX semantics

[billyc]

I second this proposal! Performance is the main problem with OMX as written. Coincidentally I was thinking of building an arrow-based Python proof of concept for this just last week. Did your proposal come out of some other conversations recently ? ..b

…

On Mon, Sep 14, 2020, 20:34 Elizabeth Sall ***@***.***> wrote: I'd like to propose that we evaluate the feasibility to support the faster Arrow <https://arrow.apache.org/>-based data format. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#37>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK4QRX53KZ44BWPBCGS4MLSFZO4VANCNFSM4RL55LMA> .

[e-lo]

Proposal based on:

1. Feedback that OMX was too slow to use in production (noted by some in the data standards learning session)
2. Use of these data formats in other contexts because of their speed benefits / hdf5 not being a standard that most people are adopting these days (that I can tell)

[pedrocamargo]

I would love for the next iteration of OMX to be based on Arrow, but is the objective of OMX to be used in production now?

[e-lo]

is the objective of OMX to be used in production now?

That's a good question for the organizing group (which is who, these days?). In practice, it is being used in production.

[pedrocamargo]

I also use it in production and made AequilibraE capable of using it as well. However, if the OMX mission changes, then I would say it would be worth it to explore other data formats to make sure we get it right.
Also, would we ask software providers to switch to the new format? Or will we support both?

[billyc]

I don't see improved performance of OMX as being a change of mission! Our tech should be useful and frictionless, to help spur adoption. Existing OMX files have a "VERSION 1" key embedded in them, precisely because we wanted the format to be changeable if the need arose. We always knew that performance of HDF5 is not great because of its slow compression library. There just weren't better alternatives at the time.

…

On Tue, Sep 15, 2020 at 3:40 AM Pedro Camargo ***@***.***> wrote: I also use it in production and made AequilibraE capable of using it as well. However, if the OMX *mission* changes, then I would say it would be worth it to explore other data formats to make sure we get it right. Also, would we ask software providers to switch to the new format? Or will we support both? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK4QRX4NQVF7URQNDQHLKTSF3AZDANCNFSM4RL55LMA> .

[gregorbj]

I think that supporting Arrow-based format and other formats in the future is probably necessary if OMX is to endure as anything more than an exchange format. The spec would have to become more abstract. One issue will be how specific should the spec be about data structure. For example it is my understanding (limited) that Arrow supports storage of tabular data in columnar format where each column can store a different data type. This is the approach that VisionEval takes. OMX stores matrix data in a matrix format. So what should the spec say in that regard. There might need to be a part of the specification to deal with each type of backend that is supported: if HDF5 how structured, if Arrow how structured, etc. Or maybe is entirely functional, identifying functions that must be supported.

[pedrocamargo]

@billyc , I was referring to using OMX for in production and not as a common format for transfer between platforms (the latter was my understanding of the mission, but I am probably mistaken and remember only part of the mission).

[bstabler]

I like this idea and I like the idea of discussing this. Anyone interested in discussing please comment on this thread and then we can brainstorm next steps - maybe a meeting to discuss, maybe a prototype, etc. Thanks!

[bstabler]

If we're thinking about a next version, let's include other potential ideas as well - more flexibility, more data types, better API conformity, CI for testing APIs, better viewers, etc.

[e-lo]

@bstabler - perhaps:

1. create a feature-request issue template
2. make a call for feature-requests (more broadly than here on github)
3. ask people to comment about their support
4. develop a backlog for next version

[jpn--]

Apparently I was not "watching" and didn't see this conversation initially. Count me in 👍

[jeabraham]

How interesting! HDF5 is primarily a disk storage format, with an option to force in-memory. Arrow is exclusively an in-memory format, right? So, the two are complimentary.

I've never been a big fan of HDF5, but don't see Arrow as a way to get away from HDF5.

Arrow sure would be nice to enable us to use higher performance libraries and not have to go through disk storage just to work in another platform or language for a bit.

[e-lo]

Arrow is exclusively an in-memory format, right?

Feather is its on-disk complement