Skip to content

feat: add Knative Kafka Broker and Trigger support - #8

Open
enchantednatures wants to merge 2 commits into
mainfrom
feat/kafka-broker-triggers
Open

feat: add Knative Kafka Broker and Trigger support#8
enchantednatures wants to merge 2 commits into
mainfrom
feat/kafka-broker-triggers

Conversation

@enchantednatures

Copy link
Copy Markdown
Owner

Summary

  • Add Knative Kafka Broker + Triggers as an eventing path alongside existing KafkaSource, enabling the Knative eventing bus with Kafka as the interchange
  • When kafka.broker.enabled: true, KafkaSources sink to the Broker and Triggers route events to services; when disabled, existing KafkaSource → Service behavior is preserved
  • Bump ksvc library chart version 0.6.0 → 0.7.0

What Changed

New Templates

  • _kafka-broker.tpl — Renders Broker CRD (eventing.knative.dev/v1 with Kafka class annotation) + ConfigMap (bootstrap servers, topic partitions/replication, auth secret, extras)
  • _kafka-trigger.tpl — Renders Trigger CRDs with full semantics: legacy filter (CloudEvents attribute matching), modern filters (exact/prefix/suffix/all/any/not/cesql), per-trigger delivery (retry/backoff), DLQ Knative Services, custom annotations (e.g. kafka.eventing.knative.dev/delivery.order: ordered)

Modified Templates

  • _kafka-source.tpl — Conditional sink: Broker ref when broker enabled, Knative Service ref when not
  • _generate.tpl — Broker/trigger defaults in applyDefaults, rendering sections for Broker+ConfigMap and Trigger loop
  • _validators.tplsource.sink only required when broker disabled; triggers require broker enabled; trigger subscriber must reference valid services key
  • _names.tpl — 4 new naming helpers: kafkaBrokerName, kafkaBrokerConfigName, kafkaTriggerName, kafkaTriggerDlqName

Tests & Examples

  • 14 broker tests, 13 trigger tests, 2 broker-sink source tests (104 total, all passing)
  • New examples/kafka-broker/ with multi-service + broker + 3 trigger patterns (unfiltered, legacy filter, modern filters with delivery ordering)

CI

  • Integration test matrix: added kafka-broker example with Knative Eventing CRD install step
  • Kubeconform skip list: added Broker,Trigger

Documentation

  • README updated: architecture diagram, naming conventions, resources rendered table, full values reference for Broker and Triggers sections, kafka-broker example listing

Values Schema

kafka:
  broker:
    enabled: true
    config:
      bootstrapServers: ["kafka.kafka.svc.cluster.local:9092"]
      topicPartitions: 10
      topicReplicationFactor: 3
  triggers:
    order-events:
      subscriber: order-processor   # references services map key
      filter:
        type: com.example.order.created
      delivery:
        retry: 10

@enchantednatures
enchantednatures force-pushed the feat/kafka-broker-triggers branch from 9347f8c to f7b4c01 Compare April 11, 2026 03:01
Add Kafka Broker as an eventing bus option alongside the existing KafkaSource
pattern. When kafka.broker is enabled, KafkaSources sink to the Broker and
Triggers route events from the Broker to Knative Services. When no broker is
defined, existing behavior (KafkaSource → Service) is preserved.

New templates:
- _kafka-broker.tpl: Broker CRD + ConfigMap (bootstrap servers, topic config, auth)
- _kafka-trigger.tpl: Trigger CRDs with full semantics (legacy filter, modern
  filters, delivery, DLQ, annotations)

Modified templates:
- _kafka-source.tpl: conditional sink (Broker vs Service)
- _generate.tpl: broker/trigger defaults and rendering loops
- _validators.tpl: broker/trigger validation rules
- _names.tpl: 4 new naming helpers

Tests: 104/104 passing (14 broker, 13 trigger, 2 broker-sink source tests)
Examples: kafka-broker example with 3 trigger patterns
CI: integration-test matrix includes kafka-broker, eventing CRD install step
Bumps chart version 0.6.0 → 0.7.0
@enchantednatures
enchantednatures force-pushed the feat/kafka-broker-triggers branch from f7b4c01 to 17bcf5f Compare April 11, 2026 03:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Knative Eventing Kafka Broker + Trigger support to the ksvc Helm library chart as an alternative eventing path alongside the existing per-topic KafkaSource → Service pattern, and bumps the chart version to 0.7.0 with updated examples/CI/tests.

Changes:

  • Introduces new Broker/Trigger Helm templates and naming helpers, plus render-time validations and values schema updates.
  • Updates KafkaSource rendering to conditionally sink to the Broker when kafka.broker.enabled: true.
  • Adds a new kafka-broker example, expands chart unit tests, and updates CI workflows to account for Broker/Trigger CRDs.

Reviewed changes

Copilot reviewed 18 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
examples/postgres-only/Chart.lock Bumps example dependency lock to ksvc 0.7.0.
examples/minimal/Chart.lock Bumps example dependency lock to ksvc 0.7.0.
examples/kafka-consumer/Chart.lock Bumps example dependency lock to ksvc 0.7.0.
examples/kafka-broker/values.yaml Adds values demonstrating Broker + multiple Triggers + delivery/DLQ options.
examples/kafka-broker/templates/common.yaml Wires the example to the library chart generator.
examples/kafka-broker/Chart.yaml Adds the new kafka-broker example chart definition.
examples/kafka-broker/Chart.lock Locks the new example chart dependency to ksvc 0.7.0.
examples/full-stack/Chart.lock Bumps example dependency lock to ksvc 0.7.0.
examples/dragonfly-cache/Chart.lock Bumps example dependency lock to ksvc 0.7.0.
charts/library/ksvc/values.schema.json Extends values schema for kafka.broker and kafka.triggers; relaxes KafkaSource sink requirement.
charts/library/ksvc/test-chart/tests/kafka-trigger_test.yaml Adds unit tests covering Trigger rendering, filters, delivery, and DLQ behavior.
charts/library/ksvc/test-chart/tests/kafka-source_test.yaml Adds unit tests for KafkaSource sink behavior with/without broker enabled.
charts/library/ksvc/test-chart/tests/kafka-broker_test.yaml Adds unit tests covering Broker + ConfigMap rendering and delivery/DLQ fields.
charts/library/ksvc/test-chart/Chart.lock Bumps test chart lock to ksvc 0.7.0.
charts/library/ksvc/templates/loader/_generate.tpl Renders Broker/ConfigMap and Trigger loop; applies defaults for broker/triggers.
charts/library/ksvc/templates/lib/_validators.tpl Adjusts sink validation for broker mode; adds trigger validation requiring broker enabled.
charts/library/ksvc/templates/lib/_names.tpl Adds naming helpers for broker/config/trigger/DLQ resources.
charts/library/ksvc/templates/classes/_kafka-trigger.tpl Adds Trigger + per-trigger DLQ Knative Service templates.
charts/library/ksvc/templates/classes/_kafka-source.tpl Updates KafkaSource sink to reference Broker when enabled.
charts/library/ksvc/templates/classes/_kafka-broker.tpl Adds Broker + ConfigMap templates for Kafka-backed eventing bus.
charts/library/ksvc/README.md Documents new Broker/Trigger capabilities, naming, and expanded values reference.
charts/library/ksvc/Chart.yaml Bumps library chart version to 0.7.0 and updates description.
charts/app/ksvc/Chart.yaml Updates app chart dependency to ksvc 0.7.0.
.gitignore Ignores app chart Chart.lock generation artifacts.
.github/workflows/lint-test.yaml Updates kubeconform skip list to include Broker/Trigger.
.github/workflows/integration-test.yaml Adds kafka-broker example to matrix and installs Knative Eventing CRDs for it.
Comments suppressed due to low confidence (1)

charts/library/ksvc/values.schema.json:542

  • values.schema.json no longer requires kafka.sources.<key>.sink, which is necessary when kafka.broker.enabled is false (the chart only fails later at template render time). Consider adding JSON Schema conditionals (if/then) so sink is required when the broker is disabled, and optional when enabled, to keep schema validation useful for consumers.
    "kafkaSourceEntry": {
      "type": "object",
      "additionalProperties": false,
      "required": ["topic"],
      "properties": {
        "topic": { "type": "string" },
        "consumerGroup": { "type": "string" },
        "bootstrapServers": {
          "type": "array",
          "minItems": 1,
          "items": { "type": "string" }
        },
        "consumers": { "type": "integer", "minimum": 1 },
        "initialOffset": {
          "type": "string",
          "enum": ["latest", "earliest"]
        },
        "sink": {
          "type": "string",
          "description": "Must reference a key in the services map"
        },
        "delivery": {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

| `kafka.sources.<key>.sink` | **Required.** Key from the `services` map | — |
| `kafka.sources.<key>.sink` | Key from the `services` map (required when broker is disabled) | — |
| `kafka.sources.<key>.consumerGroup` | Consumer group (defaults to source name) | `""` |
| `kafka.sources.<key>.bootstrapServers` | List of bootstrap servers | `[kafka.kafka.svc:9092]` |

Copilot AI Apr 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README default for kafka.sources.<key>.bootstrapServers is listed as [kafka.kafka.svc:9092], but the chart defaults in ksvc.loader.applyDefaults use kafka.kafka.svc.cluster.local:9092. Please align the documentation (or the default) so users get the correct value.

Suggested change
| `kafka.sources.<key>.bootstrapServers` | List of bootstrap servers | `[kafka.kafka.svc:9092]` |
| `kafka.sources.<key>.bootstrapServers` | List of bootstrap servers | `[kafka.kafka.svc.cluster.local:9092]` |

Copilot uses AI. Check for mistakes.
Comment on lines +207 to +208
| `kafka.broker.delivery.deadLetterSink.ref.apiVersion` | DLQ ref apiVersion | `""` |
| `kafka.broker.delivery.deadLetterSink.ref.kind` | DLQ ref kind | `""` |

Copilot AI Apr 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README lists the defaults for kafka.broker.delivery.deadLetterSink.ref.apiVersion and .kind as empty strings, but ksvc.loader.applyDefaults sets them to serving.knative.dev/v1 and Service (and the template also defaults these fields). Update the defaults in the table to match the rendered behavior, or change the defaults to actually be empty.

Suggested change
| `kafka.broker.delivery.deadLetterSink.ref.apiVersion` | DLQ ref apiVersion | `""` |
| `kafka.broker.delivery.deadLetterSink.ref.kind` | DLQ ref kind | `""` |
| `kafka.broker.delivery.deadLetterSink.ref.apiVersion` | DLQ ref apiVersion | `serving.knative.dev/v1` |
| `kafka.broker.delivery.deadLetterSink.ref.kind` | DLQ ref kind | `Service` |

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants