Create a collection of pre-built, compatibility tested, Strimzi-based Kafka Connector images

[tomncooper]

The Problem

Deploying third-party connectors on Strimzi-managed Kafka Connect can be a high friction experience for teams running event driven infrastructure on Kubernetes. Users face three options, each with with their own drawbacks:

1. Using spec.build to have Strimzi build a custom image in-cluster requires a writable container registry accessible from the cluster, and breaks down when connectors have complex transitive dependency trees or aren't published as self-contained archives. It is also not viable in locked-down production environments where strict network egress policies prevent in-cluster builds from downloading arbitrary artifacts from the internet, where security policy prohibits running build tooling (Kaniko, Buildah) inside production clusters, or where all container images must pass through an approved supply-chain pipeline (vulnerability scanning, SBOM generation, image signing) before deployment. In these environments the only permitted path is pulling a pre-approved image from an internal registry.
2. Using Kubernetes Image Volumes via spec.plugins (KEP-4639, Strimzi Proposal #102) mounts connector plugins as OCI artifacts directly into pods. This is a welcome improvement to the delivery mechanism, but it requires Kubernetes 1.33+ and, as Proposal #102 states, users are responsible for building, securing, and publishing the plugin container images and must "ensure that the files in the container image will not cause any class path conflicts." It changes how plugins get into the pod, not who does the hard work of assembling them.
3. Building a custom Docker image manually means writing a Dockerfile, resolving all connector dependencies by hand, maintaining a CI pipeline to rebuild on every Strimzi/Kafka version bump, and managing the image lifecycle entirely outside of Strimzi.

Community evidence

This is not a theoretical concern. The Strimzi community has surfaced this pain repeatedly:

• A user trying to add the Confluent Avro converter and S3 connectors found that "the version from maven are not working and the ones from confluent hub have many dependencies", ultimately falling back to a manual Docker build as a workaround.
• A user building custom connector images hit classpath conflicts with libraries in the Strimzi base image, reporting "our custom plugins sometimes conflict with runtime libraries included into base image" and "I'm struggling to find the exact versions of libraries included into an image."
• A user trying to use the Confluent S3 connector described their working setup as "a hack", having to manually hunt down kafka-connect-avro-converter JARs and place them in the right directory. The Strimzi maintainer confirmed: "I don't think there is a simpler way to do this."
• A newcomer looking for InfluxDB and Kinesis connectors couldn't find stable download URLs and expressed frustration that "it's not always clear what is basically an enterprise feature provided by Confluent and what is available as FOSS."
• A user trying to build a custom image with Debezium ran into "a lot of strange problems (for example connector status was 'running' but with 0 tasks)" and after hours of experiments gave up and switched to Confluent's Helm chart entirely.

On the upstream connector side, the packaging gap is equally visible:

• An Apache Iceberg user asked "How to build and package into jar file for Kafka Connect?" because there is no published ready-to-deploy bundle on Maven Central.
• A user deploying individual JARs instead of the full distribution hit NoClassDefFoundError because the dependency tree is too complex to assemble manually.
• A missing transitive dependency (woodstox-core) broke the connector on AWS MSK Connect after an upstream commit silently changed the dependency tree.
• There is an open request to publish the Iceberg connector to Confluent Hub, which has been blocked by CVEs in transitive dependencies (netty, aircompressor) that the project doesn't directly control.

Proposed solution

To address this pain point we could create pre-built, pre-tested connector images that layer specific connector plugins onto the standard Strimzi Kafka image. Each StreamsHub connector image would be a lightweight build project that:

• Resolves the connector's full transitive dependency tree via Maven, applying the necessary exclusions (particularly org.apache.kafka JARs already provided by the Connect runtime) to ensure classloader isolation works correctly
• Collects the resolved JARs into the standard /opt/kafka/plugins/{connector-name}/ directory structure
• Builds a container image from the Strimzi base with the plugin directory copied in
• Runs integration tests via Testcontainers that deploy the connector, produce test data, and verify end-to-end operation against the actual built image
• Publishes both a full Docker image (for spec.image on any Kubernetes version) and a minimal OCI artifact containing just the plugin JARs (for spec.plugins on Kubernetes 1.33+)

All component versions (Strimzi, Kafka, connector) would be parameterised for CI matrix testing across version combinations. Where connectors support multiple backends (e.g. cloud provider SDKs), build profiles would allow users to produce slim images containing only the dependencies they need.

StreamsHub connector images would directly address the locked-down environment problem: users could clone the image build repo, and images would then flow through the standard supply-chain pipeline (build in CI, scan, sign, mirror to internal registry) and arrive in production as pre-approved artifacts.

StreamsHub connector images would also complement Kubernetes Image Volumes rather than competing with them. Image Volumes solve the "how do I get JARs into the pod" problem. The pre-built connector images would solve the "how do I assemble compatible JARs in the first place" problem, creating the tested, versioned OCI artifacts that spec.plugins consumes. With Strimzi Proposal #118 introducing first-class support for running multiple versions of connector plugins on a single Connect cluster, pre-packaged and versioned OCI artifacts would become even more valuable as the natural unit of plugin distribution.

Candidate connectors

The following connectors would be strong candidates for pre-built StreamsHub images. Each suffers from the packaging friction described above, and would follow the same structure: a POM with connector-specific dependencies and exclusions, a minimal Dockerfile, integration tests, and dual-format publishing (Docker image + OCI artifact).

• Apache Iceberg Sink: This would be a strong first candidate. The connector has 240 runtime dependencies spanning AWS SDK, GCP, Azure, Hadoop, Hive, Parquet, ORC, and Avro, and Apache does not publish any ready-to-deploy bundle to Maven Central. Users must build the entire Iceberg monorepo from source to get a working plugin. Cloud catalog dependencies (AWS/GCP/Azure/Hive) could be managed via build profiles to keep images slim.
• Debezium (MySQL, PostgreSQL, MongoDB, SQL Server, Oracle): publishes its own debezium/connect image, but it is built on Debezium's own Kafka image, not Strimzi's. It cannot be used directly with the Strimzi operator. Debezium does publish .tar.gz plugin archives on Maven Central, so spec.build works for Debezium, but this still requires in-cluster builds and internet access. A Strimzi-native Debezium image would eliminate this friction in locked-down environments.
• JDBC Source/Sink: among the most widely deployed connectors. Multiple JDBC drivers plus the connector JARs create a non-trivial dependency matrix that users frequently get wrong.
• S3 Sink (Confluent or Apache Kafka): very common for data lake ingestion. The Confluent version requires a Confluent Hub download that Strimzi cannot natively consume; the Apache version has its own dependency tree.
• Elasticsearch/OpenSearch Sink: frequently deployed for real-time search indexing. REST client version compatibility with the target cluster is a common source of runtime LinkageError or NoClassDefFoundError.

A shared parent StreamsHub Connectors POM and/or CI workflow template could reduce per-connector boilerplate.

[tomncooper]

Also, given that Apircurio Registry now have Iceberg Catalog support there is a good integration story behind having a supported connector, catalog and maybe even UI support in the Console.