[RFC] Pluggable storage formats for the analytics engine

[LantaoJin]

The problem

The analytics engine supports one storage format, and the plugin boundary was drawn so that the
plugin owns everything from field mapping to file bytes. So format-independent work now lives inside
a format plugin: of the Parquet plugin's 8,609 non-test Java lines, about 3,240 are the
OpenSearch→Arrow adapter — 22 field implementations, the schema builder, the batch accumulator, the
buffer pool. A second format would rewrite all of it.

The sharper symptom is direction of dependency: the batch accumulator constructs its Parquet writer
and hard-references Parquet's stats, metadata, sort config and thread pool.

This blocks evaluating any format-level capability — stronger block pruning, out-of-line large
objects, reading tables the engine did not write — because "support a format" currently means "fork a
plugin".

Prior art

This is a solved problem, and the closest precedent is in-house.

Lucene's Codec SPI. Codec implements NamedSPILoader.NamedSPI; Codec.forName(String)
resolves an implementation from a name, availableCodecs() enumerates them, and implementations
register through META-INF/services. Crucially, SegmentInfos persists
si.getCodec().getName() as a string per segment and resolves it back through Codec.forName(name)
on read. That is exactly the shape this RFC proposes, one layer down: the format name is data, the
implementation is looked up, and old segments keep working because their name still resolves.
Lucene extends the same pattern to PostingsFormat and DocValuesFormat.

External example — Milvus's storage engine (Loon). It reaches five formats behind a single
name-dispatched factory, with a base class for single-file formats so that a new one implements only
"make a reader" and "make a writer". It also demonstrates the payoff of the routing half: columns are
partitioned into groups, each group stored in its own file and potentially its own format, so a query
touching few columns reads few files. It is cited here as evidence that the seam scales to five
formats, not as something to adopt wholesale.

The proposal

Change	What	Status
Shared Arrow pipeline	Move the format-agnostic OpenSearch→Arrow layer into a shared library; invert the batch accumulator to take an injected writer	built, 201 tests pass
Native format registry	A Rust Format / FormatWriter / FormatReader trait triple resolved by name, like Codec.forName	built, 5 tests pass
Field→format routing	Route fields to formats per group, giving column groups	not prototyped

A shared Arrow document pipeline

The field classes, schema builder, batch accumulator and buffer pool move to a shared library and are
re-exported. The format plugin keeps its DataFormat, engine, writer, merge strategy, settings and
stats, and depends on the shared library for the rest.

The move is mostly mechanical, with one exception that carries the actual risk: the batch accumulator
must stop constructing its writer and accept an injected one. That inversion is what makes the
pipeline shareable, and it is also the source of this proposal's one real API break — see Costs.

A native format registry

A format-seam crate defines what a format is, and a registry resolves a name to an implementation,
mirroring Codec.forName. Sketching the shape rather than the final signatures:

• a writer consumes Arrow batches and, once, finalizes into a file plus its metadata;
• a reader reports schema and row count from file metadata, and opens a projected scan that
  yields Arrow batches;
• a format names itself, states its file extension, says whether it can sort while writing, and
  hands out readers and writers.

Two decisions in that shape matter more than they look.

The FFI boundary stays outside the trait. Arrow already crosses into native code through the C
Data Interface. If the trait took raw ArrowArray* / ArrowSchema* pointers, every format
implementation would be responsible for C Data Interface safety. Importing at the FFI entry point and
passing decoded batches inward means a format author writes only safe Rust — the single largest
reduction in what a new format must get right.

The reader returns batches, not a file path. Handing the query engine a path only works for
formats it already knows how to open. Returning an Arrow stream that gets registered as a table
provider is what makes a new format queryable at all. This is the one genuinely new mechanism in
the proposal, and the one part of it that is not yet prototyped.

Field→format routing

Today a multi-format shard dual-writes whole documents: the composite document input forwards every
field to every format. Routing fields per group turns that into column groups — each group written
once, in the format that suits it. The engine already assigns a monotonic row id per document and
already remaps it when merges reorder rows, so the cross-format join coordinate exists; routing is
what makes it useful.

This is where the performance win lives. It is deliberately last, because its cost estimate depends
on the first two changes being done.

Before and after

flowchart LR
    subgraph s1["① mapping → Arrow"]
        direction TB
        T1["TODAY<br/>new format reimplements<br/>22 field impls + schema builder"]
        A1["PROPOSED<br/>shared library"]
    end
    subgraph s2["② batch accumulation"]
        direction TB
        T2["reimplements batch<br/>accumulator + buffer pool"]
        A2["shared library"]
    end
    subgraph s3["③ file write"]
        direction TB
        T3["own native writer"]
        A3["own native writer<br/>— the only real work"]
    end
    subgraph s4["④ query"]
        direction TB
        T4["own reader wiring,<br/>path-based"]
        A4["Arrow stream registered<br/>as a table provider"]
    end
    T1 --> T2 --> T3 --> T4
    A1 --> A2 --> A3 --> A4
    classDef bad fill:#fde2e2,stroke:#c0392b,color:#000
    classDef good fill:#e3f0e3,stroke:#2e7d32,color:#000
    class T1,T2,T3,T4 bad
    class A1,A2,A3,A4 good

Loading

What it costs

No performance win from the seam itself. If the answer to "do we want a second format?" is no,
this work should not happen.

A permanent API break. Injecting the writer means no constructor can default it, so every caller
must name a format; a convenience overload defaulting to Parquet would restore the very dependency
being removed. 32 test files in the prototype. Behaviour is preserved — 15 assertion lines differ
tree-wide and all 15 are type renames — but callers are not source-compatible.

JDK 25 propagates. The shared library depends on the Panama FFM binding, so every consumer of the
shared pipeline inherits that requirement.

It is a refactor on the ingest hot path. Gate: existing suites pass with no changed assertion,
plus a throughput comparison for the unchanged format.

Cross-format row alignment is argued, not tested. Independent flush cadences per format are the
case to design against.

Evidence

I have had a prototype:

prototype	result
Shared / format-specific after the split	3,241 / 5,543, against an 8,609 baseline
Net line change	+175 — the seam interfaces; the split is not a partition
Second format, native half	206 lines, complete: writes, reads, projects, reports row counts and size
Existing plugin suite	201 tests, 24 classes, 0 failures, 0 errors, 0 skipped
Per-format cost after the change	≈1,750 vs 8,609 — a 4.9× reduction

Distrust two of these appropriately: the ~1,550-line Java half of the 1,750 is a projection, and the
second format was deliberately built on a library the workspace already depends on, which separates
"does the seam hold?" from "can we take a new third-party dependency?".

Not in scope

Multi-version catalog semantics and transactional commit; a planner that reasons about which format
holds which column; changing any existing on-disk layout; migrating existing indices. No existing
format changes behaviour and no existing file changes shape.