hot reload feature proposal by Uzziee · Pull Request #1 · Uzziee/design

Uzziee · 2026-01-28T11:30:03Z

⚠️ Proposal Numbering

This PR contains proposal files that don't follow the expected format:

Issues found:

proposals/083-hot-reload-feature.md

Fix for proposals/083-hot-reload-feature.md:

git mv proposals/083-hot-reload-feature.md proposals/001-hot-reload-feature.md
# Update title: remove any old number prefix and add PR number
sed -i.bak '0,/^# /{s/^# \([0-9]\{3\}\|xxx\|nnn\|000\) - /# 1 - /; t; s/^# /# 1 - /}' proposals/001-hot-reload-feature.md && rm proposals/001-hot-reload-feature.md.bak
git add proposals/001-hot-reload-feature.md
git commit -m "Rename proposal to use PR number 1"
git push

Note: The commands above can be copied and pasted directly into your terminal. The sed command will automatically preserve your title text.

See proposals/README.md for the complete workflow.

gunnarmorling

Hey @Uzziee, a few comments/suggestions/questions inline.

gunnarmorling · 2026-01-28T12:46:37Z

@@ -0,0 +1,905 @@
+# Hot Reload Feature
+
+As of today, any changes to virtual cluster configs (addition/removal/modification) require a full restart of kroxylicious app. This proposal is to add a dynamic reload feature, which will enable operators to modify virtual cluster configurations (add/remove/modify clusters) while **maintaining service availability for unaffected clusters** without the need for full application restarts. This feature will transform Kroxylicious from a **"restart-to-configure"** system to a **"live-reconfiguration"** system


Suggested change

As of today, any changes to virtual cluster configs (addition/removal/modification) require a full restart of kroxylicious app. This proposal is to add a dynamic reload feature, which will enable operators to modify virtual cluster configurations (add/remove/modify clusters) while **maintaining service availability for unaffected clusters** without the need for full application restarts. This feature will transform Kroxylicious from a **"restart-to-configure"** system to a **"live-reconfiguration"** system

As of today, any changes to virtual cluster configs (addition/removal/modification) require a full restart of kroxylicious app. This proposal is to add a dynamic reload feature, which will enable operators to modify virtual cluster configurations (add/remove/modify clusters) while **maintaining service availability for unaffected clusters** without the need for full application restarts. This feature will transform Kroxylicious from a **"restart-to-configure"** system to a **"live-reconfiguration"** system.

gunnarmorling · 2026-01-28T12:46:44Z

@@ -0,0 +1,905 @@
+# Hot Reload Feature
+
+As of today, any changes to virtual cluster configs (addition/removal/modification) require a full restart of kroxylicious app. This proposal is to add a dynamic reload feature, which will enable operators to modify virtual cluster configurations (add/remove/modify clusters) while **maintaining service availability for unaffected clusters** without the need for full application restarts. This feature will transform Kroxylicious from a **"restart-to-configure"** system to a **"live-reconfiguration"** system


Minor: I recommend to start each sentence on a new line (or even wrap after a line of thought in a longer sentence). That way, comments can be made more targeted rather than on a larger chunk of text at once.

gunnarmorling · 2026-01-28T12:46:59Z

+
+# Part 1: Configuration Change Detection Framework
+
+With this framework, kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )


Suggested change

With this framework, kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )

With this framework, Kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )

gunnarmorling · 2026-01-28T12:47:14Z

+
+# Part 1: Configuration Change Detection Framework
+
+With this framework, kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )


So this still mentions using the file watcher for detecting changes, wheras I think the discussion is converging around using an HTTP endpoint to do so? Are you planning to update the proposal in that regard, @Uzziee?

In general, I agree that watching the file system seems not the best approach, in particular for containerized environments, where I'd expect the config to be passed in via a config map. If a config map changes, I'm not sure whether the file system watcher would pick this up (instantly)? Having a HTTP-based trigger to enforce re-reading the config seems like the right thing to do.

gunnarmorling · 2026-01-28T12:47:23Z

+
+With this framework, kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )
+
+POC PR - https://github.com/kroxylicious/kroxylicious/pull/2901


Closed by now, wanna link to the new one?

gunnarmorling · 2026-01-28T12:49:00Z

+public record ConfigurationChangeContext(
+    Configuration oldConfig,
+    Configuration newConfig,
+    List<VirtualClusterModel> oldModels,
+    List<VirtualClusterModel> newModels
+) {}


How will that accommodate changes to config parts other than clusters?

This will also include filter changes, I forgot to add it as part of this proposal as it was added later

gunnarmorling · 2026-01-28T12:49:04Z

+public record ChangeResult(
+    List<String> clustersToRemove,
+    List<String> clustersToAdd, 
+    List<String> clustersToModify
+) {}


How will that accommodate changes to config parts other than clusters?

This will also include filter changes, I forgot to add it as part of this proposal as it was added later

gunnarmorling · 2026-01-28T12:49:21Z

+
+## Flow diagram
+
+<img width="2354" height="1394" alt="Image" src="https://github.com/user-attachments/assets/19a7edba-8881-4fa7-98c8-defcdbb132da" />


This doesn't render.

Suggested change

<img width="2354" height="1394" alt="Image" src="https://github.com/user-attachments/assets/19a7edba-8881-4fa7-98c8-defcdbb132da" />

[Flow Diagram](https://github.com/user-attachments/assets/19a7edba-8881-4fa7-98c8-defcdbb132da)

gunnarmorling · 2026-01-28T12:49:32Z

+
+
+
+# Part 2: Graceful Virtual Cluster Restart


On a high level, I think the flow should be this:

Make changes to the config (e.g. remounting a K8s config map with the file)

Trigger a reload via an HTTP endpoint, which causes the file to be re-read

Identify changes to all parts of the config, exposing them in a generic way (e.g. as a list of change descriptors, each comprising the id of affected config node (cluster id, filter id, etc.) and the old and new config of that node (either one of old/new or both may be present, based on the type of change, i.e. create/update/removal)

Pass these change descriptors to a pluggable set of change handlers; initially, we might only have an handler for cluster changes, while all other changes would be ignored or even rejected)

gunnarmorling · 2026-01-28T12:50:26Z

+public class ConnectionTracker {
+
+    // Downstream connections (client → proxy)
+    private final Map<String, AtomicInteger> downstreamConnections = new ConcurrentHashMap<>();


Suggested change

private final Map<String, AtomicInteger> downstreamConnections = new ConcurrentHashMap<>();

private final ConcurrentMap<String, AtomicInteger> downstreamConnections = new ConcurrentHashMap<>();

(Here and elsewhere)

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Rewrite the hot reload proposal to focus on architectural decisions rather than implementation detail. The PR discussion has established consensus on several key points that the document didn't reflect: - Reframe around applyConfiguration(Configuration) as the core API, decoupled from trigger mechanisms (HTTP, file watcher, operator) - Remove all Java class implementations and handler chains — these belong in the code PR where they're reviewable in context - Call out remove+add with brief per-cluster downtime as deliberate - Call out all-or-nothing rollback as the initial default, consistent with startup behaviour - Move ReloadOptions to deployment-level static configuration rather than per-call parameters - Identify plugin resource tracking as a known gap with pointer to separate proposal - Flag open questions (config granularity, failure behaviour options, drain timeout configurability) - Defer trigger mechanism design as explicit future work Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk> Signed-off-by: Urjit Patel <upatel@confluent.io>

- Fix summary to read as proposed behaviour, not existing - Use "administrators" instead of "operators" for humans to avoid confusion with the Kubernetes operator process - Fix filter config examples (KMS endpoint, key selection pattern) - Clarify failure behaviour is consistent across trigger mechanisms - Note thundering herd as a known trade-off of remove+add - Fix "original proposal" to "earlier iterations" Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk> Signed-off-by: Urjit Patel <upatel@confluent.io>

…oposal This proposal introduces a mechanism for applying configuration changes to a running Kroxylicious proxy without a full restart. It defines a core `applyConfiguration(Configuration)` operation that detects changes and restarts only affected virtual clusters. Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Signed-off-by: Urjit Patel <upatel@confluent.io>

gunnarmorling reviewed Jan 28, 2026

View reviewed changes

Uzziee force-pushed the hot-reload-proposal branch from a0f6fe5 to efcebc9 Compare April 28, 2026 14:09

Uzziee and others added 11 commits May 12, 2026 13:47

hot reload feature proposal

d8683e4

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

replace file watcher with HTTP based approach

7afdec6

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Refine hot reload proposal with new policies and layers

02f44e7

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Fix wording as per comments

cb74bc4

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

refactor

8bfbc36

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Added javadoc to public applyConfiguration API

9a2b132

Signed-off-by: Urjit Patel <105218041+Uzziee@users.noreply.github.com> Signed-off-by: Urjit Patel <upatel@confluent.io>

Rename proposal to use PR number

354b0c4

Signed-off-by: Urjit Patel <upatel@confluent.io>

Updated design doc as per latest discussion

39d3638

Signed-off-by: Urjit Patel <upatel@confluent.io>

Uzziee force-pushed the hot-reload-proposal branch from efcebc9 to 39d3638 Compare May 12, 2026 08:18

Uzziee added 6 commits May 12, 2026 21:43

Added concurrency section, PR comments

7a27b62

Signed-off-by: Urjit Patel <upatel@confluent.io>

Updated Compatibility section

87b69b8

Signed-off-by: Urjit Patel <upatel@confluent.io>

Update document to reflect latest PR comments

01563db

Signed-off-by: Urjit Patel <upatel@confluent.io>

handled ConcurrentApplyException in trigger examples

6ee245c

Signed-off-by: Urjit Patel <upatel@confluent.io>

Fix indentation

ce83679

Signed-off-by: Urjit Patel <upatel@confluent.io>

Rename applyConfiguration to reconfigure and rename associated classes

dce62c4

Signed-off-by: Urjit Patel <upatel@confluent.io>

Navigation Menu

hot reload feature proposal - #1

hot reload feature proposal#1
Uzziee wants to merge 17 commits into
mainfrom
hot-reload-proposal

Uzziee commented Jan 28, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gunnarmorling left a comment

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

Uzziee Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

Uzziee Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

gunnarmorling Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,905 @@
		# Hot Reload Feature

		As of today, any changes to virtual cluster configs (addition/removal/modification) require a full restart of kroxylicious app. This proposal is to add a dynamic reload feature, which will enable operators to modify virtual cluster configurations (add/remove/modify clusters) while maintaining service availability for unaffected clusters without the need for full application restarts. This feature will transform Kroxylicious from a "restart-to-configure" system to a "live-reconfiguration" system


		# Part 1: Configuration Change Detection Framework

		With this framework, kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )


		With this framework, kroxylicious will be able to detect config file changes (using standard fileWatcher service) and using various detector interfaces, it will figure out which virtual clusters are added/removed or modified. The list of affected clusters will be then passed on to the Part 2 of this feature, where the clusters would be gracefully restarted (or rollbacked to previous stable state in case of any failures )

		POC PR - https://github.com/kroxylicious/kroxylicious/pull/2901


		## Flow diagram

		<img width="2354" height="1394" alt="Image" src="https://github.com/user-attachments/assets/19a7edba-8881-4fa7-98c8-defcdbb132da" />

	<img width="2354" height="1394" alt="Image" src="https://github.com/user-attachments/assets/19a7edba-8881-4fa7-98c8-defcdbb132da" />
	[Flow Diagram](https://github.com/user-attachments/assets/19a7edba-8881-4fa7-98c8-defcdbb132da)

	private final Map<String, AtomicInteger> downstreamConnections = new ConcurrentHashMap<>();
	private final ConcurrentMap<String, AtomicInteger> downstreamConnections = new ConcurrentHashMap<>();

Conversation

Uzziee commented Jan 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Proposal Numbering

Uh oh!

gunnarmorling left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uzziee commented Jan 28, 2026 •

edited by github-actions Bot

Loading