Repository navigation
Kafka gateway: protocol parity tracking #3560
Description
Umbrella tracking issue for the Kafka-wire-protocol gateway (gateways/), letting existing Kafka clients talk to Iggy unmodified.
Seed design + TCP listener lands in #3421.
Goal: implement enough of the Kafka wire protocol that standard Kafka producers/consumers (and admin tools) work against Iggy via a bridge, then document migration off Kafka.
Sub-issues
Foundation:
- Kafka gateway: Build Gateway to Iggy for existing Kafka users - Create a TCP Listener for Kafka users #3421 - TCP listener for Kafka clients (seed)
- Kafka gateway: Kafka gateway IggyBridge core — SDK client and stream/topic mapping #3533 - IggyBridge core: SDK client + stream/topic mapping
Phase 1 - core wire APIs + tooling:
- Kafka gateway: Kafka Metadata API (key 3) — broker and topic discovery via Iggy #3534 - Metadata API (key 3): broker/topic discovery
- Kafka gateway: Kafka Produce API (key 0) — bridge to Iggy send_messages #3535 - Produce API (key 0): bridge to
send_messages - Kafka gateway: Kafka Fetch API (key 1) — bridge to Iggy offset-based polling #3536 - Fetch API (key 1): bridge to offset-based polling
- Kafka gateway: Kafka ListOffsets API (key 2) — earliest/latest via Iggy high watermark #3537 - ListOffsets API (key 2): earliest/latest via high watermark
- Kafka gateway: Kafka CreateTopics API (key 19) — Iggy stream/topic provisioning #3538 - CreateTopics API (key 19): stream/topic provisioning
- Kafka gateway: Kafka gateway Phase 1 — docker-compose, E2E tests, and quick-start docs #3539 - Phase 1: docker-compose, E2E tests, quick-start docs
Phase 2 - consumer groups + offsets:
- Kafka gateway: Design Consumer offset persistence strategy for Kafka gateway Phase 2 #3540 - design: consumer offset persistence strategy
- Kafka gateway: Kafka FindCoordinator, JoinGroup, SyncGroup, and Heartbeat APIs #3541 - FindCoordinator, JoinGroup, SyncGroup, Heartbeat
- Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets #3542 - OffsetCommit + OffsetFetch with persisted offsets
- Kafka gateway: Kafka LeaveGroup and consumer group rebalance semantics #3543 - LeaveGroup + rebalance semantics
- Kafka gateway: Kafka gateway Phase 2 — E2E tests with consumer groups #3544 - Phase 2: E2E tests with consumer groups
Admin + extended APIs:
- Kafka gateway: Kafka transactional APIs — UNSUPPORTED_VERSION stub and idempotence docs #3545 - transactional APIs: UNSUPPORTED_VERSION stub + idempotence docs
- Kafka gateway: Kafka DeleteTopics, DescribeConfigs, and AlterConfigs admin APIs #3546 - DeleteTopics, DescribeConfigs, AlterConfigs
- Kafka gateway: Kafka DeleteRecords — trim topic to offset via Iggy retention #3547 - DeleteRecords: trim topic to offset via retention
- Kafka gateway: Kafka DescribeGroups and ListGroups admin APIs #3548 - DescribeGroups + ListGroups
Auth + migration:
- Kafka gateway: Kafka SASL authentication bridge to Iggy users #3549 - Kafka SASL auth bridge to Iggy users
- Kafka gateway: Kafka-to-Iggy migration guide — dual-write, cutover, decommission #3550 - migration guide: dual-write, cutover, decommission
Status
Stays open until boxes above are checked or de-scoped.
Activity
perf(gateways): give each Kafka producer its own Iggy client (#4502)
dina-kar commentedon Oct 21, 2026
Hi all, I'm Dinakaran from Loams, an Apache-2.0 data platform in Rust. We've chosen Iggy as our event-ingest log, and the plan is for Kafka clients (Kafka Connect/Debezium, Flink, RisingWave, Vector, Fluent Bit) to reach us through this gateway instead of a second Kafka front end of our own. So we'd like to help close the parity gaps.
These are the issues that matter most for those clients. We'd like to take them, if nobody is already on them:
Are any of these already in progress, or is there a design direction you'd like us to follow first, for #4521 especially? We'll go one PR per issue and follow CONTRIBUTING.md. I'll wait for your OK before starting.
docs(loams): plan Kafka wire compatibility in the fork