feat: add Knative Kafka Broker and Trigger support - #8
Conversation
9347f8c to
f7b4c01
Compare
Add Kafka Broker as an eventing bus option alongside the existing KafkaSource pattern. When kafka.broker is enabled, KafkaSources sink to the Broker and Triggers route events from the Broker to Knative Services. When no broker is defined, existing behavior (KafkaSource → Service) is preserved. New templates: - _kafka-broker.tpl: Broker CRD + ConfigMap (bootstrap servers, topic config, auth) - _kafka-trigger.tpl: Trigger CRDs with full semantics (legacy filter, modern filters, delivery, DLQ, annotations) Modified templates: - _kafka-source.tpl: conditional sink (Broker vs Service) - _generate.tpl: broker/trigger defaults and rendering loops - _validators.tpl: broker/trigger validation rules - _names.tpl: 4 new naming helpers Tests: 104/104 passing (14 broker, 13 trigger, 2 broker-sink source tests) Examples: kafka-broker example with 3 trigger patterns CI: integration-test matrix includes kafka-broker, eventing CRD install step Bumps chart version 0.6.0 → 0.7.0
f7b4c01 to
17bcf5f
Compare
There was a problem hiding this comment.
Pull request overview
Adds Knative Eventing Kafka Broker + Trigger support to the ksvc Helm library chart as an alternative eventing path alongside the existing per-topic KafkaSource → Service pattern, and bumps the chart version to 0.7.0 with updated examples/CI/tests.
Changes:
- Introduces new Broker/Trigger Helm templates and naming helpers, plus render-time validations and values schema updates.
- Updates KafkaSource rendering to conditionally sink to the Broker when
kafka.broker.enabled: true. - Adds a new
kafka-brokerexample, expands chart unit tests, and updates CI workflows to account for Broker/Trigger CRDs.
Reviewed changes
Copilot reviewed 18 out of 26 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/postgres-only/Chart.lock | Bumps example dependency lock to ksvc 0.7.0. |
| examples/minimal/Chart.lock | Bumps example dependency lock to ksvc 0.7.0. |
| examples/kafka-consumer/Chart.lock | Bumps example dependency lock to ksvc 0.7.0. |
| examples/kafka-broker/values.yaml | Adds values demonstrating Broker + multiple Triggers + delivery/DLQ options. |
| examples/kafka-broker/templates/common.yaml | Wires the example to the library chart generator. |
| examples/kafka-broker/Chart.yaml | Adds the new kafka-broker example chart definition. |
| examples/kafka-broker/Chart.lock | Locks the new example chart dependency to ksvc 0.7.0. |
| examples/full-stack/Chart.lock | Bumps example dependency lock to ksvc 0.7.0. |
| examples/dragonfly-cache/Chart.lock | Bumps example dependency lock to ksvc 0.7.0. |
| charts/library/ksvc/values.schema.json | Extends values schema for kafka.broker and kafka.triggers; relaxes KafkaSource sink requirement. |
| charts/library/ksvc/test-chart/tests/kafka-trigger_test.yaml | Adds unit tests covering Trigger rendering, filters, delivery, and DLQ behavior. |
| charts/library/ksvc/test-chart/tests/kafka-source_test.yaml | Adds unit tests for KafkaSource sink behavior with/without broker enabled. |
| charts/library/ksvc/test-chart/tests/kafka-broker_test.yaml | Adds unit tests covering Broker + ConfigMap rendering and delivery/DLQ fields. |
| charts/library/ksvc/test-chart/Chart.lock | Bumps test chart lock to ksvc 0.7.0. |
| charts/library/ksvc/templates/loader/_generate.tpl | Renders Broker/ConfigMap and Trigger loop; applies defaults for broker/triggers. |
| charts/library/ksvc/templates/lib/_validators.tpl | Adjusts sink validation for broker mode; adds trigger validation requiring broker enabled. |
| charts/library/ksvc/templates/lib/_names.tpl | Adds naming helpers for broker/config/trigger/DLQ resources. |
| charts/library/ksvc/templates/classes/_kafka-trigger.tpl | Adds Trigger + per-trigger DLQ Knative Service templates. |
| charts/library/ksvc/templates/classes/_kafka-source.tpl | Updates KafkaSource sink to reference Broker when enabled. |
| charts/library/ksvc/templates/classes/_kafka-broker.tpl | Adds Broker + ConfigMap templates for Kafka-backed eventing bus. |
| charts/library/ksvc/README.md | Documents new Broker/Trigger capabilities, naming, and expanded values reference. |
| charts/library/ksvc/Chart.yaml | Bumps library chart version to 0.7.0 and updates description. |
| charts/app/ksvc/Chart.yaml | Updates app chart dependency to ksvc 0.7.0. |
| .gitignore | Ignores app chart Chart.lock generation artifacts. |
| .github/workflows/lint-test.yaml | Updates kubeconform skip list to include Broker/Trigger. |
| .github/workflows/integration-test.yaml | Adds kafka-broker example to matrix and installs Knative Eventing CRDs for it. |
Comments suppressed due to low confidence (1)
charts/library/ksvc/values.schema.json:542
- values.schema.json no longer requires
kafka.sources.<key>.sink, which is necessary whenkafka.broker.enabledis false (the chart only fails later at template render time). Consider adding JSON Schema conditionals (if/then) sosinkis required when the broker is disabled, and optional when enabled, to keep schema validation useful for consumers.
"kafkaSourceEntry": {
"type": "object",
"additionalProperties": false,
"required": ["topic"],
"properties": {
"topic": { "type": "string" },
"consumerGroup": { "type": "string" },
"bootstrapServers": {
"type": "array",
"minItems": 1,
"items": { "type": "string" }
},
"consumers": { "type": "integer", "minimum": 1 },
"initialOffset": {
"type": "string",
"enum": ["latest", "earliest"]
},
"sink": {
"type": "string",
"description": "Must reference a key in the services map"
},
"delivery": {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| | `kafka.sources.<key>.sink` | **Required.** Key from the `services` map | — | | ||
| | `kafka.sources.<key>.sink` | Key from the `services` map (required when broker is disabled) | — | | ||
| | `kafka.sources.<key>.consumerGroup` | Consumer group (defaults to source name) | `""` | | ||
| | `kafka.sources.<key>.bootstrapServers` | List of bootstrap servers | `[kafka.kafka.svc:9092]` | |
There was a problem hiding this comment.
README default for kafka.sources.<key>.bootstrapServers is listed as [kafka.kafka.svc:9092], but the chart defaults in ksvc.loader.applyDefaults use kafka.kafka.svc.cluster.local:9092. Please align the documentation (or the default) so users get the correct value.
| | `kafka.sources.<key>.bootstrapServers` | List of bootstrap servers | `[kafka.kafka.svc:9092]` | | |
| | `kafka.sources.<key>.bootstrapServers` | List of bootstrap servers | `[kafka.kafka.svc.cluster.local:9092]` | |
| | `kafka.broker.delivery.deadLetterSink.ref.apiVersion` | DLQ ref apiVersion | `""` | | ||
| | `kafka.broker.delivery.deadLetterSink.ref.kind` | DLQ ref kind | `""` | |
There was a problem hiding this comment.
README lists the defaults for kafka.broker.delivery.deadLetterSink.ref.apiVersion and .kind as empty strings, but ksvc.loader.applyDefaults sets them to serving.knative.dev/v1 and Service (and the template also defaults these fields). Update the defaults in the table to match the rendered behavior, or change the defaults to actually be empty.
| | `kafka.broker.delivery.deadLetterSink.ref.apiVersion` | DLQ ref apiVersion | `""` | | |
| | `kafka.broker.delivery.deadLetterSink.ref.kind` | DLQ ref kind | `""` | | |
| | `kafka.broker.delivery.deadLetterSink.ref.apiVersion` | DLQ ref apiVersion | `serving.knative.dev/v1` | | |
| | `kafka.broker.delivery.deadLetterSink.ref.kind` | DLQ ref kind | `Service` | |
Summary
kafka.broker.enabled: true, KafkaSources sink to the Broker and Triggers route events to services; when disabled, existing KafkaSource → Service behavior is preservedWhat Changed
New Templates
_kafka-broker.tpl— Renders Broker CRD (eventing.knative.dev/v1with Kafka class annotation) + ConfigMap (bootstrap servers, topic partitions/replication, auth secret, extras)_kafka-trigger.tpl— Renders Trigger CRDs with full semantics: legacyfilter(CloudEvents attribute matching), modernfilters(exact/prefix/suffix/all/any/not/cesql), per-trigger delivery (retry/backoff), DLQ Knative Services, custom annotations (e.g.kafka.eventing.knative.dev/delivery.order: ordered)Modified Templates
_kafka-source.tpl— Conditional sink: Broker ref when broker enabled, Knative Service ref when not_generate.tpl— Broker/trigger defaults inapplyDefaults, rendering sections for Broker+ConfigMap and Trigger loop_validators.tpl—source.sinkonly required when broker disabled; triggers require broker enabled; trigger subscriber must reference valid services key_names.tpl— 4 new naming helpers:kafkaBrokerName,kafkaBrokerConfigName,kafkaTriggerName,kafkaTriggerDlqNameTests & Examples
examples/kafka-broker/with multi-service + broker + 3 trigger patterns (unfiltered, legacy filter, modern filters with delivery ordering)CI
kafka-brokerexample with Knative Eventing CRD install stepBroker,TriggerDocumentation
Values Schema