Add proposal for dynamic reloading of custom listener certificates

[acgtun]

Type of Change

• New Proposal

Description

This proposal makes the operator reload custom listener certificates (brokerCertChainAndKey) through Kafka's dynamic configuration instead of rolling all broker pods, addressing strimzi/strimzi-kafka-operator#9994.

The original blockers in #9994 no longer apply: KIP-978 (Kafka 3.8) allows keystore updates with a changed DN or SANs, and since the PEM keystore change in Strimzi 0.49 (strimzi/strimzi-kafka-operator#11447) the broker reads the certificate directly from the per-node Secret through the KubernetesSecretConfigProvider, so the operator can trigger a reload with a per-broker incrementalAlterConfigs call using the same placeholders as the static configuration.

The behavior is gated behind a new feature gate, falls back to the current rolling update on any reload failure, and starts with a proof of concept because the interaction of dynamic configs with config providers is not a documented Kafka contract (although the code path supports it, see the proposal for details including KAFKA-14136).

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• AI assistance was used to create this PR (see the Strimzi AI policy)
• The proposal uses Markdown format with one sentence per line
• BEFORE MERGING: The next free sequence number was used for the proposal and all its assets (images, etc.)
• BEFORE MERGING: The proposal index in README.md was updated

[ppatierno]

@acgtun I think we have already discussed this in other proposals. Why are you keeping opening proposals while others are still under discussion, not accepted and not implemented yet?

[scholzj]

Thanks for the proposal. Unfortunately, I do not think it is that simple because you cannot base this reliably on eventually consistent updates between the Pod annotations and the Kafka dynamic configuration updates. Especially given the completely separate parts of the code where you would update these things (StrimziPodSetController vs. StrimziPodSet resource vs. the KafkaRoller or wherever you would update it).

I originally also thought that the SAN changes are the only problem and that is why I worked on that. But after I solved that I run into the further issues.

So I'm sceptical this would work reliably. I think we need this supported in Kafka to store and return some status of the current certificate / secret.

One of my ideas was to use the Secret resource ID or generation in its configuration. E.g.:

${strimzisecrets:myproject/my-cluster-kafka-0:external-9094.key:<resourceId>}

That would be ignored by the configuration provider itself but could be used to see what the current value is. But as Kafka does not return these values, that does not work currently.

[acgtun]

Thanks for the feedback @ppatierno. My understanding was that implementation shouldn't start until a proposal is accepted, so I've been opening proposals and waiting for review rather than implementing in parallel — apologies if I got that wrong.

Right now only this one and one other proposal of mine are open, and I intend to implement both once they're accepted.

I really appreciate the time and effort you put into reviewing these and keeping Strimzi great.

[ppatierno]

My understanding was that implementation shouldn't start until a proposal is accepted, so I've been opening proposals and waiting for review rather than implementing in parallel — apologies if I got that wrong.

And that's correct. This is what I meant.

[acgtun]

My understanding was that implementation shouldn't start until a proposal is accepted, so I've been opening proposals and waiting for review rather than implementing in parallel — apologies if I got that wrong.

And that's correct. This is what I meant.

Thanks @ppatierno for the clarify