Repository navigation
Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets #3542
Description
Part of #3560.
Summary
Implement OffsetCommit (8) and OffsetFetch (9) with durable storage per #3540 design.
Motivation
Phase 2 success metrics:
- Offsets persisted across bridge restarts
kafka-console-consumer.sh --groupcommits and resumes
Depends on
- Kafka gateway: Design Consumer offset persistence strategy for Kafka gateway Phase 2 #3540 — persistence design
- Kafka gateway: Kafka FindCoordinator, JoinGroup, SyncGroup, and Heartbeat APIs #3541 — consumer group coordinator
Scope
- OffsetCommit: store
(group, topic, partition) → offset - OffsetFetch: return committed offsets on JoinGroup / consumer start
- Restart test: commit offset, restart bridge, consumer resumes
- Stub OffsetFetch for unknown group returns empty or appropriate error
Acceptance criteria
- Consumer group resumes from last committed offset after bridge restart (Option A or B)
- At-least-once on failure still documented
Activity
[-]feat(gateways): Kafka OffsetCommit and OffsetFetch with persisted consumer offsets[/-][+]Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets[/+]krishvishal commentedon Sep 30, 2026
I am picking this up.
justinmclean commentedon Oct 2, 2026
The offset design in OFFSET_STORAGE.md won't work against the current server. I assume this is this allready known?
The plan is for the gateway to save Kafka offsets in the Iggy consumer group kafka.cg. without joining that group. But the server only accepts a group offset from a client that has joined the group and owns the partition. Anyone else gets error 5009 (ConsumerGroupPartitionNotOwned). The check is fence_group_offset in core/server/src/namespace.rs.
I tested this on server 0.9.0. Saving an offset without joining fails with 5009, even when the group has no members. Reading the offset without joining works.
So if OffsetCommit is built as the doc describes, every Kafka commit through the gateway will fail.
One fix is to skip the ownership check when the group has no members. An empty group would then act as a plain offset store. Would that be acceptable on the server side?
krishvishal commentedon Oct 5, 2026
@justinmclean this PR #4392 should address the above concern. Let me know if this works.
feat(gateways): answer OffsetCommit and OffsetFetch (#4415)