Skip to content

feat(gateways): Implement CreateTopics and Metadata Kafka Keys backed by the Iggy bridge - #4258

Merged
krishvishal merged 20 commits into
apache:masterfrom
ryerraguntla:feat(gateways)/3534-3538-kafka-metadata-createtopics
Sep 28, 2026
Merged

krishvishal merged 20 commits into
apache:masterfrom
ryerraguntla:feat(gateways)/3534-3538-kafka-metadata-createtopics

Conversation

@ryerraguntla

Copy link
Copy Markdown
Contributor

Summary

Wires the Kafka gateway's CreateTopics (#3538) and Metadata (#3534) handlers to the real Iggy bridge, instead of stub responses. Combined into one PR/branch because both touch IggyBridge::get_kafka_topic —
merging up front eliminates the duplicate-implementation risk a sequential rebase would have carried, rather than just deferring it.

With IGGY_KAFKA_BRIDGE_ENABLED=true:

  • CreateTopics creates a real Iggy stream/topic, atomically reporting AlreadyExists instead of racing a check-then-create.
  • Metadata reports real topics and partition counts pulled from the bridge.

With the bridge off (default), both remain stubs as before (CreateTopics → NOT_CONTROLLER, Metadata → every topic unknown).

Closes #3534, closes #3538.
Related to #3560

What's included

  • IggyBridge::get_kafka_topic — single canonical existence/detail lookup, shared by both handlers (previously would have been two separate, differently-typed implementations).
  • IggyBridge::create_kafka_topic → TopicCreationOutcome::{Created, AlreadyExists} — atomic create-or-report-exists
  • IggyBridge::list_kafka_topics — enumerates override targets and default-stream topics for "list all," deduplicated.
  • Kafka wire error codes tightened: ERROR_INVALID_REQUEST(42), ERROR_TOPIC_ALREADY_EXISTS(36), ERROR_INVALID_CONFIG(40), ERROR_REQUEST_TIMED_OUT(7). Removed the incorrect ERROR_INVALID_REPLICA_ASSIGNMENT(39).
  • Bridge fan-out hardening, applied identically to both handlers: dedup requested topic names before any bridge call, MAX_BRIDGE_BACKED_TOPICS = 100 cap (rejected wholesale as INVALID_REQUEST, no bridge call at
    all above the cap), one wall-clock deadline over the request's aggregate bridge work (CreateTopics uses the wire timeout_ms, clamped [1s, 30s]; Metadata uses a fixed 20s, no timeout_ms field in its supported
    versions).
  • Metadata: fixed v0 isAllTopics() semantics (topics == null || (topics.isEmpty() && version == 0)); added a post-bridge-call, pre-encode frame-size check (response_would_exceed_frame_size) since partition-count
    expansion (up to 1000/topic) isn't priced by the pre-decode element-count bound in bounds_guard.rs.

Test plan

  • cargo build -p iggy-gateway-kafka --all-targets — clean.
  • cargo fmt --all, cargo sort --no-format --workspace — clean.
  • cargo clippy -p iggy-gateway-kafka --all-features --all-targets -- -D warnings — no issues.
  • cargo test -p iggy-gateway-kafka — 332 passed, 16 suites (includes new create_topics_real_bridge_tests.rs, metadata_real_bridge_tests.rs, and 4 new list_kafka_topics_* cases appended to
    bridge_iggy_integration_tests.rs).
  • ./scripts/ci/{trailing-whitespace,trailing-newline,license-headers,markdownlint}.sh — clean.
  • git diff --check — no stray whitespace.

AI Usage

If AI tools were used, please answer:

  1. Which tools? Claude code for implementation , Cursor for review and validation
  2. Scope of usage? Implementation , review and validation
  3. How did you verify the generated code works correctly? Test Suite and Code walkthrough
  4. Can you explain every line of the code if asked? Yes

Wires the Kafka gateway's CreateTopics (apache#3538) and Metadata (apache#3534) handlers to the real Iggy bridge.
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 22, 2026
@ryerraguntla ryerraguntla changed the title Feat(gateways)/3534 3538 kafka metadata createtopics feat(gateways): Implement CreateTopics and Metadata Kafka Keys backed by the Iggy bridge Sep 22, 2026
@codecov

codecov Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.59%. Comparing base (8d0e28d) to head (0a0f871).

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #4258       +/-   ##
=============================================
- Coverage     87.56%   27.59%   -59.97%     
- Complexity     1575     1576        +1     
=============================================
  Files          1284     1281        -3     
  Lines        225457   181924    -43533     
  Branches     188821   145288    -43533     
=============================================
- Hits         197416    50203   -147213     
- Misses        23314   129947   +106633     
+ Partials       4727     1774     -2953     
Components Coverage Δ
Rust Core 12.56% <ø> (-76.20%) ⬇️
Java SDK 68.70% <ø> (+0.01%) ⬆️
C# SDK 77.41% <ø> (-0.02%) ⬇️
Python SDK 90.97% <ø> (ø)
PHP SDK 85.67% <ø> (ø)
Node SDK 96.43% <ø> (+1.68%) ⬆️
Go SDK 70.05% <ø> (ø)
see 691 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread gateways/kafka/src/protocol/handlers/create_topics.rs
Comment thread gateways/kafka/src/protocol/handlers/metadata.rs Outdated
Comment thread gateways/kafka/src/bridge/error.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/metadata.rs
Comment thread gateways/kafka/src/protocol/handlers/create_topics.rs Outdated
@ryerraguntla

Copy link
Copy Markdown
Contributor Author

/ready

Comment thread gateways/kafka/src/protocol/handlers/metadata.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/create_topics.rs Outdated
Comment thread gateways/kafka/src/bridge/iggy_bridge/topics.rs Outdated
…f ssh://github.com/ryerraguntla/iggy into feat(gateways)/3534-3538-kafka-metadata-createtopics
@ryerraguntla

Copy link
Copy Markdown
Contributor Author

/ready

@krishvishal krishvishal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I request changes for the 20 warnings below.
The largest are the 100-name Metadata cap, the batch deadlines that drop finished results, and the configs rejection that breaks Kafka Connect.

Comment thread gateways/kafka/src/protocol/handlers/metadata.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/create_topics.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/metadata.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/metadata.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/create_topics.rs Outdated
Comment thread gateways/kafka/src/bridge/iggy_bridge/topics.rs Outdated
Comment thread gateways/kafka/src/bridge/iggy_bridge/topics.rs
Comment thread gateways/kafka/src/protocol/handlers/metadata.rs Outdated
Comment thread gateways/kafka/src/protocol/handlers/create_topics.rs
Comment thread gateways/kafka/README.md Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 24, 2026
@ryerraguntla

Copy link
Copy Markdown
Contributor Author

/ready

1 similar comment
@ryerraguntla

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Sep 25, 2026
@numinnex

Copy link
Copy Markdown
Contributor

@ryerraguntla resolve the conflict and from my side it's good to go.

numinnex
numinnex previously approved these changes Sep 25, 2026
krishvishal
krishvishal previously approved these changes Sep 25, 2026
@ryerraguntla
ryerraguntla dismissed stale reviews from krishvishal and numinnex via 6920b66 September 25, 2026 23:28
@krishvishal
krishvishal merged commit 15a49d0 into apache:master Sep 28, 2026
70 checks passed
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants