Skip to content

[Java] Add zero-copy ingestIpcBatch for Arrow Flight#289

Draft
danilonajkov-db wants to merge 1 commit into
mainfrom
java-zer-copy
Draft

[Java] Add zero-copy ingestIpcBatch for Arrow Flight#289
danilonajkov-db wants to merge 1 commit into
mainfrom
java-zer-copy

Conversation

@danilonajkov-db
Copy link
Copy Markdown
Contributor

@danilonajkov-db danilonajkov-db commented May 18, 2026

What changes are proposed in this pull request?

Adds ZerobusArrowStream.ingestIpcBatch(byte[]) — a zero-copy ingestion path that forwards
pre-serialized Arrow IPC bytes directly into the Flight wire format, skipping the
deserialize → re-serialize round-trip performed by ingestBatch(VectorSchemaRoot). Wires
through a new nativeIngestIpcBatch JNI export to ZerobusArrowStream::ingest_ipc_batch
in the Rust core.

Why: the existing ingestBatch path materializes a RecordBatch JNI-side only to
immediately re-encode it for Flight. Callers that already hold IPC bytes (e.g. from Kafka,
files, network sources) had no way to skip this. The new method gives them one.

Constraint: incompatible with streams configured with ipcCompression — raw IPC bytes
wouldn't match the codec. Surfaces as a ZerobusException.

How is this tested?

  • mvn test -Dtest=ArrowIntegrationTest against staging — 18 tests pass, including 3 new
    ones for this feature: happy-path ingest, compression-rejection, null/empty input.
  • cargo clippy -p zerobus-jni -- -D warnings, cargo fmt --check, cargo test --lib
    clean.
  • mvn spotless:check, full Java unit tests — clean.

@danilonajkov-db danilonajkov-db changed the title changes [Java] Add zero-copy ingestIpcBatch for Arrow Flight May 18, 2026
@danilonajkov-db danilonajkov-db self-assigned this May 18, 2026
@danilonajkov-db danilonajkov-db marked this pull request as draft May 18, 2026 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant