From 8a2f387af625c2fb53ff215cc47dfcf4f9195f43 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 14 May 2026 16:17:12 +1200 Subject: [PATCH 01/15] Draft system test framework proposal Introduces layered abstractions (ProxyScenario, FilterSpec, ProxyFixture, ProxyHandle) that separate test intent from deployment mechanism, enabling deployment-agnostic feature tests and test-first development. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/xxx-system-test-framework.md | 394 +++++++++++++++++++++++++ 1 file changed, 394 insertions(+) create mode 100644 proposals/xxx-system-test-framework.md diff --git a/proposals/xxx-system-test-framework.md b/proposals/xxx-system-test-framework.md new file mode 100644 index 00000000..37491312 --- /dev/null +++ b/proposals/xxx-system-test-framework.md @@ -0,0 +1,394 @@ +# xxx - System Test Framework + +## Summary + +Introduce a layered abstraction for system tests that separates test intent (what the proxy should do) from deployment mechanism (how the proxy is stood up). A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. Feature tests become portable across deployment mechanisms — the same test runs against an operator-managed proxy, a manifest-managed proxy, or a downstream distribution — and cheap enough to write before the production code, as a specification. + +## Current Situation + +The system test suite covers the right things at the right level. Assertions in `RecordEncryptionST`, `AuthorizationST`, and `EntityIsolationST` test feature behaviour cleanly. `OperatorChangeDetectionST` tests operator reconciliation behaviour. The abstraction breaks down only in setup. + +Every feature test class has a private `deployXxx()` method that reimplements the same builder/template pattern against the operator's CRD types. Adding a new optional parameter (e.g. `ExperimentalKmsConfig`) requires touching every one of them. Timing workarounds are scattered across test classes with comments pointing at unresolved issues. The convergence question — "is the proxy actually serving the configuration I just applied?" — is answered by ad hoc polling in each test class rather than by a framework-level contract. + +This setup cost has a second-order effect: system tests are written after features merge, delegated to QE because they are too expensive for a developer to include in a feature PR. The test framework is the bottleneck, not the assertions. + +The fix required is narrow: a thin setup layer that hides the deployment machinery without changing the assertions at all. + +## Motivation + +A system test is a precise, executable description of intended behaviour: deploy the proxy in configuration Y (Given), perturb it (When), assert the result (Then). If the framework makes that cheap to express, writing the system test first is feature-level TDD. The test fails until the production code makes it pass, and passing it is the definition of done. + +This model has already been validated in the codebase: the checksum annotation mechanism in the operator was built test-first. The system test described what the operator should do; the implementation followed. + +Three problems prevent this from being the norm: + +1. **Setup cost**: expressing "a proxy with this filter" requires navigating CRD builders, template classes, namespace management, and ingress configuration. The ceremony dwarfs the test. + +2. **No convergence contract**: the framework does not define when the proxy is ready. Each test class independently polls for readiness, with varying strategies and varying reliability. + +3. **Operator coupling**: every test implicitly requires the operator. Feature tests — which care only that a correctly-configured proxy is serving traffic — cannot run without the full operator installation. This conflates feature correctness with operator correctness and prevents fast local iteration. + +Addressing these enables: + +- **Test-first development**: a developer writing a new filter can write a failing system test as the first commit of their feature branch, without reading framework documentation or asking QE for help. +- **Deployment-agnostic feature tests**: the same test runs against an operator-managed proxy, a manifest-managed proxy, or a Helm installation, with no changes to the test body. +- **Reliable convergence**: `proxyFixture.apply()` is a blocking call with a defined contract — when it returns, the proxy is serving the requested configuration. Manual polling disappears from test classes. +- **A TCK for downstream distributions**: downstream distributors can implement `ProxyFixture` and run the upstream feature test suite against their distribution without forking the test module. + +## Proposal + +### The Primary Seam + +The framework needs one organising question answered for every test class: **what is this test covering?** + +- **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a correctly-configured proxy exists and is serving traffic. They must not care how the proxy was deployed. + +- **Operator tests** — does the operator detect a configuration change and trigger a rolling restart? These tests are explicitly about the operator's reconciliation behaviour. They require the operator to be present. + +This is the primary design axis. Everything else — how resources are applied, how convergence is waited for, what assertions are available — follows from it. + +A test in the first group is runnable against an operator-managed deployment, a manifest-managed deployment, or any other deployment mechanism. A test in the second group is necessarily operator-only, is tagged accordingly, and skips gracefully when the operator is not present. + +### `ProxyScenario` — Intent Without Deployment + +A plain Java value object describing what configuration the proxy should have. No knowledge of namespaces, CRD templates, or deployment mechanism. + +```java +ProxyScenario scenario = ProxyScenario.builder() + .withUpstream(clusterName) + .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) + .build(); + +ProxyScenario scenario = ProxyScenario.builder() + .withUpstream(clusterName) + .withFilter(new RecordEncryptionFilterSpec(testKmsFacade) + .withExperimentalConfig(config)) + .withDownstreamTls(tls) + .build(); +``` + +### `FilterSpec` — The Filter DSL + +`FilterSpec` is an interface for expressing which filter the proxy should run and how it should be configured — in terms of the filter's purpose, not its deployment mechanics. No template classes, no filter type names, no namespaces visible to the test author. + +First-party filter types ship named implementations: + +```java +new RecordEncryptionFilterSpec(testKmsFacade) +new SimpleTransformFilterSpec("foo", "bar") +``` + +Custom filter authors implement `FilterSpec` directly for their own filter type, using whatever config model their filter defines. This is the same extension point a developer would use when writing a system test before the filter exists: the `FilterSpec` implementation is the specification. + +An escape hatch avoids the need for a dedicated class when one is not justified: + +```java +new RawFilterSpec("com.example.MyFilter", new MyFilterConfig(...)) +``` + +### `ProxyFixture` — Application and Convergence + +The fixture translates a `ProxyScenario` into running infrastructure, blocks until the proxy has converged, and returns a `ProxyHandle`. + +```java +interface ProxyFixture { + ProxyHandle apply(ProxyScenario scenario); +} +``` + +This is an explicit call — not magic JUnit injection — because the test author needs to understand that `apply()` is a blocking operation that includes convergence waiting. Hiding it behind injection would obscure the framework's most important contract. + +Two implementations cover the primary deployment mechanisms: + +**`OperatorProxyFixture`**: applies the Kroxylicious CRDs via Server-Side Apply, then waits for observable convergence signals in sequence: +1. The pod template's `kroxylicious.io/referent-checksum` annotation changes from its previous value — the operator has seen and processed the update. +2. The Deployment reaches stable state — updated replicas are ready and serving. + +**`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. + +Both implementations use Server-Side Apply. Neither requires `createOrUpdate` branching or `resourceVersion` management. + +**A note on convergence**: the framework waits for the best observable signal, not a guarantee. There is an inherent gap between "the operator updated the Deployment" and "the new pods are handling traffic." Tests that need stronger guarantees use extension points (described below). The `ProxyFixture` contract is: when `apply()` returns, the proxy is serving the requested configuration to the best observable precision. + +### `ProxyHandle` — A Token of Convergence + +The only way to obtain a `ProxyHandle` is through `ProxyFixture.apply()`. This means a test cannot accidentally interact with the proxy before convergence has been waited for. + +```java +interface ProxyHandle { + String bootstrap(); + String bootstrap(ClientLocation location); + ProxyHandle reconfigure(ProxyScenario scenario); + void waitForRestart(); +} +``` + +`bootstrap()` defaults to `ClientLocation.ON_CLUSTER`. Tests using off-cluster clients call `bootstrap(ClientLocation.OFF_CLUSTER)` to obtain the externally accessible address; the fixture provides the right value for the deployment. + +`waitForRestart()` is on `ProxyHandle` rather than on any capability because restarting the proxy is meaningful across all fixture types — on Kubernetes the fixture observes the Deployment rollout; on bare metal the fixture manages the process restart directly. The concept is universal; the mechanism is fixture-specific. + +### Injection Model + +`ProxyFixture` is injected by the JUnit extension at class scope — it is an environment configuration concern, long-lived, with no timing implications. `ProxyHandle` is always obtained explicitly by calling `proxyFixture.apply()` in the test body. This call is blocking and includes convergence waiting; making it explicit ensures the test author understands the contract. + +```java +ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible +``` + +Tests tagged `@Operator` have `OperatorCapability` injected as a test parameter by the JUnit extension. The tag declares the requirement; if the active fixture does not support the operator, the extension skips the test before it runs. + +```java +@Operator +@Test +void testChecksumChanges(OperatorCapability operator) { + ProxyHandle proxy = proxyFixture.apply(scenario); + String before = operator.currentChecksum(); + proxy.reconfigure(updatedScenario); + operator.waitForChecksumChange(before); +} +``` + +| Concept | How obtained | Reason | +|---|---|---| +| `ProxyFixture` | Injected (class-scoped) | Environment config, no timing implications | +| `OperatorCapability` | Injected for `@Operator` tests | Tag declares requirement; skip handled by extension | +| `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | + +### `OperatorCapability` — Operator-Observable State + +`OperatorCapability` is narrow and precise. It contains only things that require the operator to be present — state that does not exist in a manifest-managed deployment: + +```java +interface OperatorCapability { + String currentChecksum(); + void waitForChecksumChange(String previousChecksum); + List currentStatusConditions(); +} +``` + +`waitForRestart()` is intentionally absent — it lives on `ProxyHandle` because it applies to all fixture types. Other capabilities (`MetricsCapability`, `TlsCapability`) follow the same injection pattern for their respective tags. + +### `KafkaClient` Abstraction + +The existing `KafkaClient` interface with its multiple implementations (StrimziTestClient, KcatClient, KafClient, PythonTestClient) — selected at runtime via environment variable — is the right shape and largely works. The gap is off-cluster support. All current implementations run as Kubernetes jobs; an off-cluster client (an embedded Java client in the test JVM, or a client process on a bare metal host) has no namespace and no container image. + +The current interface conflates the core produce/consume contract with Kubernetes-specific machinery: + +```java +KafkaClient inNamespace(String namespace); +String getImage(); +void preloadImage(); +``` + +These move to a `KubernetesClientCapability`, following the same extension point pattern as `OperatorCapability`: + +```java +interface KafkaClient { + ExecResult produceMessages(...); + List consumeMessages(...); + Optional as(Class capability); +} + +interface KubernetesClientCapability { + KafkaClient inNamespace(String namespace); + String getImage(); + void preloadImage(); +} +``` + +An off-cluster embedded Java client implements `KafkaClient` only. The existing pod-based clients implement both. Code that pre-pulls images or sets a namespace calls `as(KubernetesClientCapability.class)` and skips gracefully if absent. + +**Bootstrap address pairing**: an off-cluster client needs an externally accessible bootstrap address, not the cluster-internal Service DNS. This pairs naturally with `ProxyHandle.bootstrap(ClientLocation)` — the framework wires client location to bootstrap address at test setup time. The test author calls `proxy.bootstrap()` and receives the right address for the client that is configured. + +### Cluster Environment + +The framework must be runnable across three meaningfully different environments: + +| Environment | Characteristics | +|---|---| +| Minikube / local K8s | Local dev path. No OLM. Limited networking. Fast iteration. | +| Vanilla remote K8s | CI path. OLM optional. LoadBalancer or NodePort ingress. | +| OpenShift (OCP) | OLM native. Routes instead of Ingress. Security Context Constraints. | + +The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. An `OperatorProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. + +Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same mechanism as `@Operator` and `@AdmissionWebhook` tags, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. + +### Deployment Tests + +A separate, lightweight test class per supported install method confirms that the installation mechanism produces a working proxy. Each test is a single scenario: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. + +This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the feature test suite; the deployment test only asserts that the installation mechanism puts the proxy in a state where features can run. + +Each deployment test has a single reason to fail: the consumer did not see the transformed value. Every possible installation failure collapses into that one observable. No separate assertions per failure mode are needed or wanted; they all manifest identically, and the test name tells you which installation mechanism failed. + +| Install method | Fixture | File config mechanism | +|---|---|---| +| Operator (Helm) | `OperatorProxyFixture` | Kubernetes Secret mount | +| Operator (OLM) | `OlmProxyFixture` | Kubernetes Secret mount | +| Manifest (Helm, no operator) | `ManifestProxyFixture` | Kubernetes Secret mount | +| Manifest (Kustomize / raw YAML) | `ManifestProxyFixture` | Kubernetes Secret mount | +| Sidecar injection (webhook) | `SidecarProxyFixture` | Kubernetes Secret mount | +| Bare metal | `BareMetalProxyFixture` | File written to local path | + +### Admission Webhook Tests + +There are two distinct classes of test for the admission webhook. + +**Sidecar injection as a deployment path**: `SidecarProxyFixture` creates a pod with the injection annotation, waits for the webhook to mutate it, waits for the sidecar to be ready, and returns a `ProxyHandle`. Feature tests run against it unchanged. The deployment smoke test for this path is the same file-based filter scenario as every other install method — if the sidecar is injected and the plugin loads, the installation mechanism works. + +**Webhook behaviour tests**: a separate category that asserts on the Kubernetes API interception layer rather than on proxy behaviour. These tests ask questions that do not produce a `ProxyHandle`: does the webhook inject into pods with annotation X but not Y? Does it produce a valid pod spec? What happens when the webhook is unavailable and `failurePolicy: Ignore`? + +These are tagged `@AdmissionWebhook` and skip automatically when the webhook is not installed — the same pattern as `@Operator` tests. + +### `ProxyFixture` as a TCK Extension Point + +`ProxyFixture` is a plain Java interface with no upstream-specific dependencies in its signature. A downstream distributor can implement `ProxyFixture` without forking the upstream test module. + +With a downstream fixture in place, they can run the upstream feature test suite against their distribution: + +```bash +mvn test -Pfixture=com.example.downstream.MyProxyFixture +``` + +The feature test assertions are upstream's; the deployment is downstream's. Upstream maintains the definition of "correct behaviour"; downstream validates that their distribution satisfies it. This is the TCK model — the same seam that separates operator from manifest also separates upstream from downstream. + +There is no separate downstream framework to maintain. The extension point is the interface. + +### What Feature Tests Look Like + +The before/after comparison illustrates the effect of the abstraction on a representative test. + +**Before** (current `RecordEncryptionST`): + +```java +private void deployPortIdentifiesNodeWithRecordEncryptionFilter( + TestKmsFacade testKmsFacade, ExperimentalKmsConfig config) { + String filterName = KROXYLICIOUS_ENCRYPTION_FILTER_NAME + "-" + + testKmsFacade.getKmsServiceClass().getSimpleName().toLowerCase(); + kroxylicious = new KroxyliciousBuilder() + .withNamespace(Constants.KROXYLICIOUS_NAMESPACE) + .withKafkaProxy(KroxyliciousKafkaProxyTemplates + .defaultKafkaProxyCR(KROXYLICIOUS_PROXY_SIMPLE_NAME, 1).build()) + .withKafkaProxyIngress(KroxyliciousKafkaProxyIngressTemplates + .defaultKafkaProxyIngressCR(KROXYLICIOUS_INGRESS_CLUSTER_IP, + KROXYLICIOUS_PROXY_SIMPLE_NAME).build()) + .withKafkaService(KroxyliciousKafkaClusterRefTemplates + .defaultKafkaClusterRefCR(clusterName).build()) + .addKafkaProtocolFilter(KroxyliciousFilterTemplates + .kroxyliciousRecordEncryptionFilter(KROXYLICIOUS_NAMESPACE, + filterName, testKmsFacade, config).build()) + .withVirtualKafkaCluster(KroxyliciousVirtualKafkaClusterTemplates + .virtualKafkaClusterWithFilterCR(clusterName, + KROXYLICIOUS_PROXY_SIMPLE_NAME, clusterName, + KROXYLICIOUS_INGRESS_CLUSTER_IP, + List.of(filterName)).build()) + .build(); + kroxylicious.createOrUpdateResources(); +} +``` + +**After**: + +```java +@Test +void ensureClusterHasEncryptedMessage(String namespace) { + testKmsFacade.getTestKekManager().generateKek(KEK_PREFIX + topicName); + + ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() + .withUpstream(clusterName) + .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) + .build()); + + KafkaSteps.createTopic(namespace, topicName, proxy.bootstrap(), 1, 1); + KroxyliciousSteps.produceMessages(namespace, topicName, proxy.bootstrap(), MESSAGE, 1); + + var consumed = KroxyliciousSteps.consumeMessageFromKafkaCluster(...); + assertThat(consumed).allMatch(r -> !r.getPayload().contains(MESSAGE)); +} +``` + +The test contains only the Given/When/Then relevant to record encryption. It works against both an operator-managed and a manifest-managed proxy. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. + +### What Operator Tests Look Like + +**Before** (current `OperatorChangeDetectionST`): + +```java +@Test +void shouldUpdateWhenFilterConfigurationChanges(String namespace) { + resourceManager.createOrUpdateResourceFromBuilderWithWait(arbitraryFilterBuilder); + deployPortIdentifiesNodeWithFilters(namespace, kafkaClusterName, List.of("arbitrary-filter")); + + var originalChecksum = getInitialChecksum(namespace); // ~30 lines of polling + var replacementConfig = Map.of("transformation", "Replacing", + "transformationConfig", Map.of("findPattern", "foo", "replacementValue", "updated")); + + resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> { + current.getSpec().setConfigTemplate(replacementConfig); + }); + + assertDeploymentUpdated(namespace, originalChecksum); // polls checksum annotation +} +``` + +**After**: + +```java +@Operator +@Test +void shouldUpdateWhenFilterConfigurationChanges(OperatorCapability operator) { + ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() + .withUpstream(clusterName) + .withFilter(new SimpleTransformFilterSpec("foo", "bar")) + .build()); + + String before = operator.currentChecksum(); + + resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> + current.getSpec().setConfigTemplate(replacementConfig)); + + operator.waitForChecksumChange(before); +} +``` + +`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `operator.currentChecksum()` is called against stable state. The direct `resourceManager.replaceResourceWithRetries` call is intentionally visible — these tests exist to prove the operator detects and responds to mutations made outside the fixture. + +## Affected/Not Affected Projects + +**Affected:** +- **kroxylicious-systemtest**: the test framework module. New abstractions (`ProxyScenario`, `FilterSpec`, `ProxyFixture`, `ProxyHandle`, `OperatorCapability`) are introduced here. Existing test classes are migrated incrementally. +- **kroxylicious-operator**: no code changes, but operator-managed system tests are rewritten to use `OperatorProxyFixture` and `OperatorCapability`. + +**Not affected:** +- **kroxylicious-proxy (runtime)**: no production code changes. The framework abstracts over the proxy; it does not change it. +- **kroxylicious-api**: the filter SPI is unaffected. +- **kroxylicious-kms and plugin modules**: no changes needed. + +## Compatibility + +This proposal introduces new framework abstractions alongside the existing code. Existing tests continue to work throughout the migration — the new layer wraps the existing `Kroxylicious` class internally. No test assertions change; only setup code is replaced. + +The `ProxyFixture` interface is designed for extension. Downstream distributors can implement it without depending on upstream internals. Once published, the `ProxyFixture`, `ProxyScenario`, and `ProxyHandle` interfaces become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. + +## Rejected Alternatives + +### Magic JUnit injection of `ProxyHandle` + +We considered having the JUnit extension inject `ProxyHandle` directly as a test parameter (similar to how `ProxyFixture` is injected), with convergence waiting happening transparently during parameter resolution. This hides the most important contract in the framework — that `apply()` is a blocking operation — behind invisible lifecycle callbacks. The test author would not see when convergence happens, making it harder to reason about test timing and harder to debug when convergence fails. The explicit `proxyFixture.apply()` call keeps the blocking operation visible. + +### Unified `ProxyFixture.apply()` for both feature and operator tests + +We considered having operator tests use `proxyFixture.apply()` for all mutations, including the mid-test configuration changes that operator tests need to assert on. However, operator tests exist specifically to prove that the operator detects mutations made outside the fixture — bypassing the fixture for the mid-test mutation is the point of the test. Routing those mutations through the fixture would test the fixture's update path, not the operator's reconciliation behaviour. + +### Merging `OperatorCapability` into `ProxyHandle` + +We considered putting operator-specific methods (checksum observation, status conditions) directly on `ProxyHandle`, with runtime exceptions for unsupported operations. This conflates two concerns: `ProxyHandle` represents a converged proxy regardless of deployment mechanism, while `OperatorCapability` represents state that only exists in operator-managed deployments. Keeping them separate means `ProxyHandle` has no methods that might throw "not supported" — every method on it is meaningful for every fixture type. + +### Per-environment fixture implementations + +We considered separate fixture classes for each cluster environment (e.g. `MinikubeOperatorProxyFixture`, `OCPOperatorProxyFixture`). This creates a combinatorial explosion of fixture classes and pushes environment-specific logic into class hierarchies. Environment differences are narrower than deployment-mechanism differences: the same `OperatorProxyFixture` can create a Route on OCP and a LoadBalancer Service on vanilla K8s based on a constructor-time environment flag. The fixture class models the deployment mechanism; environment variation is configuration within that class. + +### Abstract base class instead of `ProxyFixture` interface + +We considered providing an abstract base class with shared convergence-waiting logic rather than a plain interface. This would simplify fixture implementations at the cost of coupling them to the upstream base class. A downstream distributor who needs different convergence behaviour (e.g. polling a proprietary health endpoint) would need to override carefully or bypass the base class entirely. The plain interface is a cleaner extension point — shared convergence utilities can be offered as composition rather than inheritance. \ No newline at end of file From 199653e7de3133bd2f32a245973095887db3c34d Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 14 May 2026 16:31:45 +1200 Subject: [PATCH 02/15] Decouple OperatorCapability from checksum mechanism OperatorCapability now models the operator's externally observable state via generation-based reconciliation observation rather than the checksum annotation. Tests that assert on specific operator mechanisms (e.g. checksum change detection) observe resource state directly. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/xxx-system-test-framework.md | 30 +++++++++++++++----------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/proposals/xxx-system-test-framework.md b/proposals/xxx-system-test-framework.md index 37491312..eb910fa8 100644 --- a/proposals/xxx-system-test-framework.md +++ b/proposals/xxx-system-test-framework.md @@ -100,9 +100,7 @@ This is an explicit call — not magic JUnit injection — because the test auth Two implementations cover the primary deployment mechanisms: -**`OperatorProxyFixture`**: applies the Kroxylicious CRDs via Server-Side Apply, then waits for observable convergence signals in sequence: -1. The pod template's `kroxylicious.io/referent-checksum` annotation changes from its previous value — the operator has seen and processed the update. -2. The Deployment reaches stable state — updated replicas are ready and serving. +**`OperatorProxyFixture`**: applies the Kroxylicious CRDs via Server-Side Apply, then waits for observable convergence signals — the operator has reconciled the resource (e.g. `status.observedGeneration` matches `metadata.generation`) and the Deployment has reached stable state with updated replicas ready and serving. The specific convergence signals may evolve as the operator matures (see [OperatorCapability](#operatorcapability--operator-observable-state)). **`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. @@ -140,11 +138,11 @@ Tests tagged `@Operator` have `OperatorCapability` injected as a test parameter ```java @Operator @Test -void testChecksumChanges(OperatorCapability operator) { +void testReconciliation(OperatorCapability operator) { ProxyHandle proxy = proxyFixture.apply(scenario); - String before = operator.currentChecksum(); + long generation = operator.observedGeneration(filterResource); proxy.reconfigure(updatedScenario); - operator.waitForChecksumChange(before); + operator.waitForReconciliation(filterResource, generation); } ``` @@ -156,16 +154,20 @@ void testChecksumChanges(OperatorCapability operator) { ### `OperatorCapability` — Operator-Observable State -`OperatorCapability` is narrow and precise. It contains only things that require the operator to be present — state that does not exist in a manifest-managed deployment: +`OperatorCapability` models the externally observable state of the operator — what an observer can determine by inspecting Kubernetes resource status, not by knowing the operator's internal mechanisms. ```java interface OperatorCapability { - String currentChecksum(); - void waitForChecksumChange(String previousChecksum); + long observedGeneration(HasMetadata resource); + void waitForReconciliation(HasMetadata resource, long sinceGeneration); List currentStatusConditions(); } ``` +`observedGeneration()` returns the generation the operator has most recently reconciled for a given resource. `waitForReconciliation()` blocks until the operator has reconciled past the specified generation. The fixture implementation can use whatever signal backs this — `status.observedGeneration`, annotation changes, or lifecycle state transitions — without affecting the test. + +Tests that assert on specific operator mechanisms (e.g. `OperatorChangeDetectionST` verifying that checksum annotations change in response to referent mutations) should observe resource state directly rather than through `OperatorCapability`. Those tests exist to prove a specific mechanism works; abstracting it away would hide the thing being tested. + `waitForRestart()` is intentionally absent — it lives on `ProxyHandle` because it applies to all fixture types. Other capabilities (`MetricsCapability`, `TlsCapability`) follow the same injection pattern for their respective tags. ### `KafkaClient` Abstraction @@ -343,16 +345,18 @@ void shouldUpdateWhenFilterConfigurationChanges(OperatorCapability operator) { .withFilter(new SimpleTransformFilterSpec("foo", "bar")) .build()); - String before = operator.currentChecksum(); + long generation = operator.observedGeneration(arbitraryFilter); resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> current.getSpec().setConfigTemplate(replacementConfig)); - operator.waitForChecksumChange(before); + operator.waitForReconciliation(arbitraryFilter, generation); } ``` -`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `operator.currentChecksum()` is called against stable state. The direct `resourceManager.replaceResourceWithRetries` call is intentionally visible — these tests exist to prove the operator detects and responds to mutations made outside the fixture. +`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `operator.observedGeneration()` is called against stable state — no polling required to establish a baseline. The direct `resourceManager.replaceResourceWithRetries` call is intentionally visible — these tests exist to prove the operator detects and responds to mutations made outside the fixture. + +A test like `OperatorChangeDetectionST` that specifically asserts the checksum annotation mechanism works would not use `OperatorCapability` for this — it would read the pod template annotation directly, since the annotation is the thing being tested. ## Affected/Not Affected Projects @@ -383,7 +387,7 @@ We considered having operator tests use `proxyFixture.apply()` for all mutations ### Merging `OperatorCapability` into `ProxyHandle` -We considered putting operator-specific methods (checksum observation, status conditions) directly on `ProxyHandle`, with runtime exceptions for unsupported operations. This conflates two concerns: `ProxyHandle` represents a converged proxy regardless of deployment mechanism, while `OperatorCapability` represents state that only exists in operator-managed deployments. Keeping them separate means `ProxyHandle` has no methods that might throw "not supported" — every method on it is meaningful for every fixture type. +We considered putting operator-specific methods (reconciliation observation, status conditions) directly on `ProxyHandle`, with runtime exceptions for unsupported operations. This conflates two concerns: `ProxyHandle` represents a converged proxy regardless of deployment mechanism, while `OperatorCapability` represents state that only exists in operator-managed deployments. Keeping them separate means `ProxyHandle` has no methods that might throw "not supported" — every method on it is meaningful for every fixture type. ### Per-environment fixture implementations From 519ccdf7f9350c012598624fa47b41e09c995ac3 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 15 May 2026 11:12:13 +1200 Subject: [PATCH 03/15] Introduce KubernetesClient as general-purpose resource access OperatorCapability handles convergence and deployment agnosticism. Tests that assert on specific resource state (e.g. checksum annotations) use an injected KubernetesClient to observe resources directly, keeping the two concerns separate. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/xxx-system-test-framework.md | 31 +++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/proposals/xxx-system-test-framework.md b/proposals/xxx-system-test-framework.md index eb910fa8..239c9e97 100644 --- a/proposals/xxx-system-test-framework.md +++ b/proposals/xxx-system-test-framework.md @@ -150,6 +150,7 @@ void testReconciliation(OperatorCapability operator) { |---|---|---| | `ProxyFixture` | Injected (class-scoped) | Environment config, no timing implications | | `OperatorCapability` | Injected for `@Operator` tests | Tag declares requirement; skip handled by extension | +| `KubernetesClient` | Injected (test parameter) | General-purpose access to cluster resource state | | `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | ### `OperatorCapability` — Operator-Observable State @@ -166,7 +167,25 @@ interface OperatorCapability { `observedGeneration()` returns the generation the operator has most recently reconciled for a given resource. `waitForReconciliation()` blocks until the operator has reconciled past the specified generation. The fixture implementation can use whatever signal backs this — `status.observedGeneration`, annotation changes, or lifecycle state transitions — without affecting the test. -Tests that assert on specific operator mechanisms (e.g. `OperatorChangeDetectionST` verifying that checksum annotations change in response to referent mutations) should observe resource state directly rather than through `OperatorCapability`. Those tests exist to prove a specific mechanism works; abstracting it away would hide the thing being tested. +Tests that assert on specific operator mechanisms — such as `OperatorChangeDetectionST` verifying that checksum annotations change in response to referent mutations — use `OperatorCapability` for convergence and deployment agnosticism, but observe the specific resource state via an injected `KubernetesClient`. The `KubernetesClient` is a general-purpose facility for reading cluster state; it is not operator-specific. This keeps `OperatorCapability` focused on the generic reconciliation contract while giving tests direct access to the resources they need to assert on: + +```java +@Operator +@Test +void shouldUpdateChecksumWhenFilterConfigChanges( + OperatorCapability operator, KubernetesClient kubeClient) { + ProxyHandle proxy = proxyFixture.apply(scenario); + String beforeChecksum = readChecksumAnnotation(kubeClient, deploymentName); + long generation = operator.observedGeneration(arbitraryFilter); + + resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> + current.getSpec().setConfigTemplate(replacementConfig)); + + operator.waitForReconciliation(arbitraryFilter, generation); + assertThat(readChecksumAnnotation(kubeClient, deploymentName)) + .isNotEqualTo(beforeChecksum); +} +``` `waitForRestart()` is intentionally absent — it lives on `ProxyHandle` because it applies to all fixture types. Other capabilities (`MetricsCapability`, `TlsCapability`) follow the same injection pattern for their respective tags. @@ -339,24 +358,26 @@ void shouldUpdateWhenFilterConfigurationChanges(String namespace) { ```java @Operator @Test -void shouldUpdateWhenFilterConfigurationChanges(OperatorCapability operator) { +void shouldUpdateWhenFilterConfigurationChanges( + OperatorCapability operator, KubernetesClient kubeClient) { ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() .withUpstream(clusterName) .withFilter(new SimpleTransformFilterSpec("foo", "bar")) .build()); + String beforeChecksum = readChecksumAnnotation(kubeClient, deploymentName); long generation = operator.observedGeneration(arbitraryFilter); resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> current.getSpec().setConfigTemplate(replacementConfig)); operator.waitForReconciliation(arbitraryFilter, generation); + assertThat(readChecksumAnnotation(kubeClient, deploymentName)) + .isNotEqualTo(beforeChecksum); } ``` -`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `operator.observedGeneration()` is called against stable state — no polling required to establish a baseline. The direct `resourceManager.replaceResourceWithRetries` call is intentionally visible — these tests exist to prove the operator detects and responds to mutations made outside the fixture. - -A test like `OperatorChangeDetectionST` that specifically asserts the checksum annotation mechanism works would not use `OperatorCapability` for this — it would read the pod template annotation directly, since the annotation is the thing being tested. +`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so both `operator.observedGeneration()` and `readChecksumAnnotation()` are called against stable state — no polling required to establish a baseline. `OperatorCapability` handles convergence and deployment agnosticism (the test works whether the operator was installed via OLM or Helm); the `KubernetesClient` gives direct access to the resource state being asserted on. The direct `resourceManager.replaceResourceWithRetries` call is intentionally visible — these tests exist to prove the operator detects and responds to mutations made outside the fixture. ## Affected/Not Affected Projects From d743cae7257ccd5a68f69012c36ea5d320efc0fa Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 15 May 2026 16:27:03 +1200 Subject: [PATCH 04/15] Separate test modules, Installer interface, and CrdProxyFixture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three distinct test categories become separate modules with different compile-time dependencies: systemtest-feature (no K8s dependency), systemtest-operator (K8s client and CRD types), and systemtest-installer (one test per install method). All three are TCK-consumable. Installer is a public interface — the primary downstream extension point. CrdProxyFixture replaces OperatorProxyFixture to reflect that the fixture uses the CRD API, not the operator's internals. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/xxx-system-test-framework.md | 239 ++++++++++++++++--------- 1 file changed, 158 insertions(+), 81 deletions(-) diff --git a/proposals/xxx-system-test-framework.md b/proposals/xxx-system-test-framework.md index 239c9e97..442f6712 100644 --- a/proposals/xxx-system-test-framework.md +++ b/proposals/xxx-system-test-framework.md @@ -39,15 +39,27 @@ Addressing these enables: ### The Primary Seam -The framework needs one organising question answered for every test class: **what is this test covering?** +The framework needs one organising question answered for every test class: **what is this test covering?** The answer determines which module the test belongs to — not which tag it carries, but which compile-time dependencies it has. -- **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a correctly-configured proxy exists and is serving traffic. They must not care how the proxy was deployed. +Three categories of system test have fundamentally different concerns: -- **Operator tests** — does the operator detect a configuration change and trigger a rolling restart? These tests are explicitly about the operator's reconciliation behaviour. They require the operator to be present. +- **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a correctly-configured proxy exists and is serving traffic. They must not care how the proxy was deployed. They have no Kubernetes dependency. -This is the primary design axis. Everything else — how resources are applied, how convergence is waited for, what assertions are available — follows from it. +- **Operator tests** — does the operator detect a configuration change and trigger a rolling restart? These tests are explicitly about the operator's reconciliation behaviour. They interact with Kubernetes resources directly and depend on the Kubernetes client. -A test in the first group is runnable against an operator-managed deployment, a manifest-managed deployment, or any other deployment mechanism. A test in the second group is necessarily operator-only, is tagged accordingly, and skips gracefully when the operator is not present. +- **Installer tests** — does a specific installation method (OLM, Helm, kustomize, standalone) produce a working proxy? These tests are about the installation mechanism, not the proxy features. Each test knows its installer. + +These are not tags on a single test suite — they are separate modules with different compile-time dependencies: + +| Module | Depends on | Kubernetes dependency | Portable across fixtures | +|---|---|---|---| +| `systemtest-feature` | `ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec` | None | Yes — runs against any fixture | +| `systemtest-operator` | Above + `KubernetesCapability`, CRD types, K8s client | Yes | No — requires CRD-based fixture | +| `systemtest-installer` | Above + `KubernetesCapability`, `Installer` | Yes (except standalone) | No — one test per installer | + +Feature tests do not import Kubernetes types. They cannot accidentally depend on CRDs, namespaces, or client libraries. The module boundary enforces this at compile time, not by convention. + +All three modules are consumable as a TCK. A downstream distributor runs feature tests to prove their distribution satisfies the proxy's behavioural contract, installer tests to prove their installation method works, and operator tests to prove operator reconciliation works with their installation. ### `ProxyScenario` — Intent Without Deployment @@ -98,15 +110,55 @@ interface ProxyFixture { This is an explicit call — not magic JUnit injection — because the test author needs to understand that `apply()` is a blocking operation that includes convergence waiting. Hiding it behind injection would obscure the framework's most important contract. -Two implementations cover the primary deployment mechanisms: +Fixture implementations span two independent concerns: how the infrastructure is installed (CRDs, RBAC, operator Deployment) and how proxy instances are deployed. These concerns are separated by composing a `ProxyFixture` with an `Installer`. + +### `Installer` — Infrastructure Installation + +`Installer` handles getting the operator, CRDs, RBAC rules, and ServiceAccounts into the cluster. It is a **public interface** — the primary extension point for downstream distributors, who typically vary only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed. + +```java +interface Installer { + void install(); + void uninstall(); +} +``` + +Upstream ships implementations for each supported installation method: + +| Installer | What it installs | +|---|---| +| `ManifestInstaller` | Operator via kustomize/raw manifests (upstream default) | +| `HelmInstaller` | Operator via Helm chart | +| `OlmInstaller` | Operator via OLM catalog | + +A downstream distributor implements `Installer` for their distribution and composes it with upstream's fixture — no need to reimplement proxy deployment or convergence logic. + +### Fixture Implementations -**`OperatorProxyFixture`**: applies the Kroxylicious CRDs via Server-Side Apply, then waits for observable convergence signals — the operator has reconciled the resource (e.g. `status.observedGeneration` matches `metadata.generation`) and the Deployment has reached stable state with updated replicas ready and serving. The specific convergence signals may evolve as the operator matures (see [OperatorCapability](#operatorcapability--operator-observable-state)). +Fixtures compose an `Installer` (where applicable) with proxy deployment logic: + +```java +// CRD-backed: installer puts operator in cluster, fixture deploys via CRDs +new CrdProxyFixture(new ManifestInstaller()) +new CrdProxyFixture(new OlmInstaller()) +new CrdProxyFixture(new MyDownstreamInstaller()) + +// Manifest-backed: deploys proxy directly, no operator +new ManifestProxyFixture() + +// Standalone: local Java process, no Kubernetes +new StandaloneProxyFixture() +``` -**`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. +**`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the operator into the cluster; the fixture applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. -Both implementations use Server-Side Apply. Neither requires `createOrUpdate` branching or `resourceVersion` management. +**`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. -**A note on convergence**: the framework waits for the best observable signal, not a guarantee. There is an inherent gap between "the operator updated the Deployment" and "the new pods are handling traffic." Tests that need stronger guarantees use extension points (described below). The `ProxyFixture` contract is: when `apply()` returns, the proxy is serving the requested configuration to the best observable precision. +**`StandaloneProxyFixture`**: starts the proxy as a local Java process with a generated configuration file, waits for the port to be ready, and returns a `ProxyHandle` with a localhost bootstrap. No Kubernetes, no installer, no namespaces. `KubernetesCapability` is not available for tests running against this fixture. + +Kubernetes fixtures use Server-Side Apply. Neither `CrdProxyFixture` nor `ManifestProxyFixture` requires `createOrUpdate` branching or `resourceVersion` management. + +**A note on convergence**: the framework waits for the best observable signal, not a guarantee. There is an inherent gap between "the operator updated the Deployment" and "the new pods are handling traffic." The `ProxyFixture` contract is: when `apply()` returns, the proxy is serving the requested configuration to the best observable precision. ### `ProxyHandle` — A Token of Convergence @@ -125,7 +177,7 @@ interface ProxyHandle { `waitForRestart()` is on `ProxyHandle` rather than on any capability because restarting the proxy is meaningful across all fixture types — on Kubernetes the fixture observes the Deployment rollout; on bare metal the fixture manages the process restart directly. The concept is universal; the mechanism is fixture-specific. -### Injection Model +### Injection Model and Tags `ProxyFixture` is injected by the JUnit extension at class scope — it is an environment configuration concern, long-lived, with no timing implications. `ProxyHandle` is always obtained explicitly by calling `proxyFixture.apply()` in the test body. This call is blocking and includes convergence waiting; making it explicit ensures the test author understands the contract. @@ -133,61 +185,59 @@ interface ProxyHandle { ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible ``` -Tests tagged `@Operator` have `OperatorCapability` injected as a test parameter by the JUnit extension. The tag declares the requirement; if the active fixture does not support the operator, the extension skips the test before it runs. +**Tags as skip conditions**: `@Operator` and `@AdmissionWebhook` are tags that declare a test's infrastructure requirements. If the required component is not present, the extension skips the test before it runs. Tags declare requirements — they do not select fixtures. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). -```java -@Operator -@Test -void testReconciliation(OperatorCapability operator) { - ProxyHandle proxy = proxyFixture.apply(scenario); - long generation = operator.observedGeneration(filterResource); - proxy.reconfigure(updatedScenario); - operator.waitForReconciliation(filterResource, generation); -} -``` +**`KubernetesCapability`**: any test running on Kubernetes can have `KubernetesCapability` injected as a test parameter. This provides general-purpose access to the cluster environment — the namespace the proxy was deployed into and a `KubernetesClient` for observing resource state. Tests running on bare metal do not have `KubernetesCapability` available. | Concept | How obtained | Reason | |---|---|---| | `ProxyFixture` | Injected (class-scoped) | Environment config, no timing implications | -| `OperatorCapability` | Injected for `@Operator` tests | Tag declares requirement; skip handled by extension | -| `KubernetesClient` | Injected (test parameter) | General-purpose access to cluster resource state | +| `KubernetesCapability` | Injected for Kubernetes-deployed tests | Namespace and client access for resource observation | | `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | -### `OperatorCapability` — Operator-Observable State +### `KubernetesCapability` — Cluster Environment Access -`OperatorCapability` models the externally observable state of the operator — what an observer can determine by inspecting Kubernetes resource status, not by knowing the operator's internal mechanisms. +`KubernetesCapability` provides access to the Kubernetes environment the proxy was deployed into. It is not operator-specific — it is available for any Kubernetes-backed fixture (operator, manifest, sidecar). ```java -interface OperatorCapability { - long observedGeneration(HasMetadata resource); - void waitForReconciliation(HasMetadata resource, long sinceGeneration); - List currentStatusConditions(); +interface KubernetesCapability { + String namespace(); + KubernetesClient client(); } ``` -`observedGeneration()` returns the generation the operator has most recently reconciled for a given resource. `waitForReconciliation()` blocks until the operator has reconciled past the specified generation. The fixture implementation can use whatever signal backs this — `status.observedGeneration`, annotation changes, or lifecycle state transitions — without affecting the test. +The namespace is managed by the fixture. Each `apply()` call deploys into a namespace the fixture controls; the test discovers it through the capability rather than supplying it. This keeps `ProxyScenario` free of deployment concerns while giving tests the access they need for resource observation and client operations. -Tests that assert on specific operator mechanisms — such as `OperatorChangeDetectionST` verifying that checksum annotations change in response to referent mutations — use `OperatorCapability` for convergence and deployment agnosticism, but observe the specific resource state via an injected `KubernetesClient`. The `KubernetesClient` is a general-purpose facility for reading cluster state; it is not operator-specific. This keeps `OperatorCapability` focused on the generic reconciliation contract while giving tests direct access to the resources they need to assert on: +### Tags and Component Requirements -```java -@Operator -@Test -void shouldUpdateChecksumWhenFilterConfigChanges( - OperatorCapability operator, KubernetesClient kubeClient) { - ProxyHandle proxy = proxyFixture.apply(scenario); - String beforeChecksum = readChecksumAnnotation(kubeClient, deploymentName); - long generation = operator.observedGeneration(arbitraryFilter); - - resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> - current.getSpec().setConfigTemplate(replacementConfig)); - - operator.waitForReconciliation(arbitraryFilter, generation); - assertThat(readChecksumAnnotation(kubeClient, deploymentName)) - .isNotEqualTo(beforeChecksum); -} +`@Operator` and `@AdmissionWebhook` are skip tags — they declare that a test requires a specific component and cause the extension to skip the test if that component is not present. The operator itself is infrastructure: the extension uses the configured `Installer` to deploy it before operator-tagged tests run. + +The test does not interact with the operator directly. It interacts with the proxy (via `ProxyHandle`) and with Kubernetes resources (via `KubernetesCapability`). The operator is the mechanism that makes the proxy appear in response to CRDs; the test observes the result, not the mechanism. + +### Fixture and Installer Selection + +Fixture and installer are selected independently via system properties: + +```bash +# Default upstream: operator installed via manifests, proxy deployed via CRDs +mvn test -Dfixture=crd -Dinstaller=manifest + +# OLM installation +mvn test -Dfixture=crd -Dinstaller=olm + +# Downstream custom installer, upstream fixture +mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller + +# Manifest-managed proxy (no operator) +mvn test -Dfixture=manifest + +# Standalone +mvn test -Dfixture=standalone ``` -`waitForRestart()` is intentionally absent — it lives on `ProxyHandle` because it applies to all fixture types. Other capabilities (`MetricsCapability`, `TlsCapability`) follow the same injection pattern for their respective tags. +The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `-Dinstaller` is not specified, the fixture uses its default (`ManifestInstaller` for operator fixtures). Standalone and manifest fixtures do not take an installer. + +If a test in `systemtest-operator` runs but the active fixture is `ManifestProxyFixture` or `StandaloneProxyFixture`, the test skips — the operator is not present, and the fixture cannot satisfy the requirement. ### `KafkaClient` Abstraction @@ -201,7 +251,7 @@ String getImage(); void preloadImage(); ``` -These move to a `KubernetesClientCapability`, following the same extension point pattern as `OperatorCapability`: +These move to a `KubernetesClientCapability`: ```java interface KafkaClient { @@ -231,7 +281,7 @@ The framework must be runnable across three meaningfully different environments: | Vanilla remote K8s | CI path. OLM optional. LoadBalancer or NodePort ingress. | | OpenShift (OCP) | OLM native. Routes instead of Ingress. Security Context Constraints. | -The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. An `OperatorProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. +The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. An `CrdProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same mechanism as `@Operator` and `@AdmissionWebhook` tags, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. @@ -243,14 +293,15 @@ This test is deliberately minimal — it is not a feature matrix. Its purpose is Each deployment test has a single reason to fail: the consumer did not see the transformed value. Every possible installation failure collapses into that one observable. No separate assertions per failure mode are needed or wanted; they all manifest identically, and the test name tells you which installation mechanism failed. -| Install method | Fixture | File config mechanism | -|---|---|---| -| Operator (Helm) | `OperatorProxyFixture` | Kubernetes Secret mount | -| Operator (OLM) | `OlmProxyFixture` | Kubernetes Secret mount | -| Manifest (Helm, no operator) | `ManifestProxyFixture` | Kubernetes Secret mount | -| Manifest (Kustomize / raw YAML) | `ManifestProxyFixture` | Kubernetes Secret mount | -| Sidecar injection (webhook) | `SidecarProxyFixture` | Kubernetes Secret mount | -| Bare metal | `BareMetalProxyFixture` | File written to local path | +| Install method | Fixture | Installer | File config mechanism | +|---|---|---|---| +| CRD (manifests) | `CrdProxyFixture` | `ManifestInstaller` | Kubernetes Secret mount | +| CRD (Helm) | `CrdProxyFixture` | `HelmInstaller` | Kubernetes Secret mount | +| CRD (OLM) | `CrdProxyFixture` | `OlmInstaller` | Kubernetes Secret mount | +| Manifest (Helm, no operator) | `ManifestProxyFixture` | — | Kubernetes Secret mount | +| Manifest (Kustomize / raw YAML) | `ManifestProxyFixture` | — | Kubernetes Secret mount | +| Sidecar injection (webhook) | `SidecarProxyFixture` | `ManifestInstaller` | Kubernetes Secret mount | +| Standalone | `StandaloneProxyFixture` | — | File written to local path | ### Admission Webhook Tests @@ -262,19 +313,25 @@ There are two distinct classes of test for the admission webhook. These are tagged `@AdmissionWebhook` and skip automatically when the webhook is not installed — the same pattern as `@Operator` tests. -### `ProxyFixture` as a TCK Extension Point +### TCK Extension Points -`ProxyFixture` is a plain Java interface with no upstream-specific dependencies in its signature. A downstream distributor can implement `ProxyFixture` without forking the upstream test module. +The framework provides two public interfaces for downstream extensibility: `ProxyFixture` and `Installer`. -With a downstream fixture in place, they can run the upstream feature test suite against their distribution: +Most downstream distributors differ only in how the operator is installed — their own OLM catalog, their own Helm chart, a different RBAC configuration. These distributors implement `Installer` and compose it with upstream's `CrdProxyFixture`, inheriting all proxy deployment and convergence logic: ```bash -mvn test -Pfixture=com.example.downstream.MyProxyFixture +mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller ``` -The feature test assertions are upstream's; the deployment is downstream's. Upstream maintains the definition of "correct behaviour"; downstream validates that their distribution satisfies it. This is the TCK model — the same seam that separates operator from manifest also separates upstream from downstream. +Distributors with fundamentally different deployment models (e.g. a custom orchestrator, a managed service) implement `ProxyFixture` directly. Both interfaces have no upstream-specific dependencies in their signatures. -There is no separate downstream framework to maintain. The extension point is the interface. +All three test modules are consumable as a TCK: + +- **`systemtest-feature`**: downstream proves their distribution satisfies the proxy's behavioural contract — features work regardless of installation method. +- **`systemtest-installer`**: downstream proves their installation method produces a working system — their OLM catalog installs correctly, their Helm chart renders valid resources. +- **`systemtest-operator`**: downstream proves operator reconciliation works with their installation — change detection, status conditions, rolling restarts all function correctly. + +Upstream maintains the definition of correct behaviour across all three modules; downstream provides the `Installer` (and optionally the `ProxyFixture`) that adapts the tests to their distribution. ### What Feature Tests Look Like @@ -358,32 +415,38 @@ void shouldUpdateWhenFilterConfigurationChanges(String namespace) { ```java @Operator @Test -void shouldUpdateWhenFilterConfigurationChanges( - OperatorCapability operator, KubernetesClient kubeClient) { +void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() .withUpstream(clusterName) .withFilter(new SimpleTransformFilterSpec("foo", "bar")) .build()); - String beforeChecksum = readChecksumAnnotation(kubeClient, deploymentName); - long generation = operator.observedGeneration(arbitraryFilter); + String beforeChecksum = readChecksumAnnotation(kube.client(), kube.namespace()); - resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> - current.getSpec().setConfigTemplate(replacementConfig)); + kube.client().resources(KafkaProtocolFilter.class) + .inNamespace(kube.namespace()) + .withName(filterName) + .edit(current -> { + current.getSpec().setConfigTemplate(replacementConfig); + return current; + }); - operator.waitForReconciliation(arbitraryFilter, generation); - assertThat(readChecksumAnnotation(kubeClient, deploymentName)) - .isNotEqualTo(beforeChecksum); + await().until( + () -> readChecksumAnnotation(kube.client(), kube.namespace()), + not(equalTo(beforeChecksum))); } ``` -`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so both `operator.observedGeneration()` and `readChecksumAnnotation()` are called against stable state — no polling required to establish a baseline. `OperatorCapability` handles convergence and deployment agnosticism (the test works whether the operator was installed via OLM or Helm); the `KubernetesClient` gives direct access to the resource state being asserted on. The direct `resourceManager.replaceResourceWithRetries` call is intentionally visible — these tests exist to prove the operator detects and responds to mutations made outside the fixture. +`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `readChecksumAnnotation()` is called against stable state — no polling required to establish the baseline. This test lives in the `systemtest-operator` module, which has compile-time access to `KubernetesCapability` and the CRD types. The test does not interact with the operator directly — it observes the operator's effect on Kubernetes resources. The resource mutation is intentionally direct — this test exists to prove the operator detects and responds to changes made outside the fixture. ## Affected/Not Affected Projects **Affected:** -- **kroxylicious-systemtest**: the test framework module. New abstractions (`ProxyScenario`, `FilterSpec`, `ProxyFixture`, `ProxyHandle`, `OperatorCapability`) are introduced here. Existing test classes are migrated incrementally. -- **kroxylicious-operator**: no code changes, but operator-managed system tests are rewritten to use `OperatorProxyFixture` and `OperatorCapability`. +- **systemtest-feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec`). No Kubernetes dependency. +- **systemtest-operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. +- **systemtest-installer**: new module. One deployment smoke test per supported installation method. +- **kroxylicious-systemtest (framework)**: the shared framework module providing `ProxyFixture`, `Installer`, `ProxyScenario`, `ProxyHandle`, `KubernetesCapability`, and fixture implementations. +- **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest-operator`. **Not affected:** - **kroxylicious-proxy (runtime)**: no production code changes. The framework abstracts over the proxy; it does not change it. @@ -394,7 +457,7 @@ void shouldUpdateWhenFilterConfigurationChanges( This proposal introduces new framework abstractions alongside the existing code. Existing tests continue to work throughout the migration — the new layer wraps the existing `Kroxylicious` class internally. No test assertions change; only setup code is replaced. -The `ProxyFixture` interface is designed for extension. Downstream distributors can implement it without depending on upstream internals. Once published, the `ProxyFixture`, `ProxyScenario`, and `ProxyHandle` interfaces become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. +The `ProxyFixture` and `Installer` interfaces are designed for extension. Downstream distributors typically implement `Installer` and compose it with upstream fixtures; distributors with fundamentally different deployment models implement `ProxyFixture` directly. Once published, `ProxyFixture`, `Installer`, `ProxyScenario`, and `ProxyHandle` become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. ## Rejected Alternatives @@ -406,13 +469,27 @@ We considered having the JUnit extension inject `ProxyHandle` directly as a test We considered having operator tests use `proxyFixture.apply()` for all mutations, including the mid-test configuration changes that operator tests need to assert on. However, operator tests exist specifically to prove that the operator detects mutations made outside the fixture — bypassing the fixture for the mid-test mutation is the point of the test. Routing those mutations through the fixture would test the fixture's update path, not the operator's reconciliation behaviour. -### Merging `OperatorCapability` into `ProxyHandle` +### `OperatorCapability` as a test-facing API + +We considered exposing an `OperatorCapability` interface to tests, providing methods like `observedGeneration()`, `waitForReconciliation()`, and `currentStatusConditions()`. This would give operator tests a typed API for interacting with the operator's observable state. -We considered putting operator-specific methods (reconciliation observation, status conditions) directly on `ProxyHandle`, with runtime exceptions for unsupported operations. This conflates two concerns: `ProxyHandle` represents a converged proxy regardless of deployment mechanism, while `OperatorCapability` represents state that only exists in operator-managed deployments. Keeping them separate means `ProxyHandle` has no methods that might throw "not supported" — every method on it is meaningful for every fixture type. +On closer examination, operator tests do not need to interact with the operator — they observe its effects on Kubernetes resources. The change detection test reads a checksum annotation; the status condition test reads a resource's status. Both are Kubernetes API observations, not operator interactions. `KubernetesCapability` provides everything these tests need. The operator is infrastructure the extension manages; the `@Operator` tag ensures it is present and selects the right fixture. Making the operator invisible to the test keeps `ProxyHandle` deployment-agnostic and avoids an abstraction that doesn't carry its weight. + +### Merging `KubernetesCapability` into `ProxyHandle` + +We considered putting Kubernetes-specific methods (namespace, client access) directly on `ProxyHandle`. This would make `ProxyHandle` Kubernetes-aware, breaking its deployment-agnostic contract — a bare metal `ProxyHandle` has no namespace. Keeping them separate means every method on `ProxyHandle` is meaningful for every fixture type. + +### Fixture selection via ServiceLoader + +We considered using `ServiceLoader` to discover `ProxyFixture` implementations automatically from the classpath. This creates ambiguity when multiple fixture implementations are present (e.g. both OLM and Helm operator fixtures) and makes test runs non-deterministic. An explicit system property or Maven profile provides clear, reproducible fixture selection and composes naturally with CI matrix builds. ### Per-environment fixture implementations -We considered separate fixture classes for each cluster environment (e.g. `MinikubeOperatorProxyFixture`, `OCPOperatorProxyFixture`). This creates a combinatorial explosion of fixture classes and pushes environment-specific logic into class hierarchies. Environment differences are narrower than deployment-mechanism differences: the same `OperatorProxyFixture` can create a Route on OCP and a LoadBalancer Service on vanilla K8s based on a constructor-time environment flag. The fixture class models the deployment mechanism; environment variation is configuration within that class. +We considered separate fixture classes for each cluster environment (e.g. `MinikubeCrdProxyFixture`, `OCPCrdProxyFixture`). This creates a combinatorial explosion of fixture classes and pushes environment-specific logic into class hierarchies. Environment differences are narrower than deployment-mechanism differences: the same `CrdProxyFixture` can create a Route on OCP and a LoadBalancer Service on vanilla K8s based on a constructor-time environment flag. The fixture class models the deployment mechanism; environment variation is configuration within that class. + +### `Installer` as extension-internal + +We considered hiding `Installer` as an extension-internal interface, with downstream distributors implementing the full `ProxyFixture`. However, downstream typically varies only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed or converged. Making `Installer` public lets downstream write a single class and compose it with upstream's fixture, inheriting all proxy deployment and convergence logic. Forcing them to reimplement `ProxyFixture` for a difference that is entirely in the installation dimension wastes effort and risks divergence. ### Abstract base class instead of `ProxyFixture` interface From d22a9a95259cdd4af44198c4d9f53b3377651f1a Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 15 May 2026 16:38:47 +1200 Subject: [PATCH 05/15] Clarify that tests never instantiate fixtures or installers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixture/installer composition is extension-internal, driven by system properties. Installer tests use a single smoke test — the test does not vary between installers, CI provides the matrix. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/xxx-system-test-framework.md | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/proposals/xxx-system-test-framework.md b/proposals/xxx-system-test-framework.md index 442f6712..fc7b1e01 100644 --- a/proposals/xxx-system-test-framework.md +++ b/proposals/xxx-system-test-framework.md @@ -135,20 +135,9 @@ A downstream distributor implements `Installer` for their distribution and compo ### Fixture Implementations -Fixtures compose an `Installer` (where applicable) with proxy deployment logic: +Tests never instantiate fixtures or installers — the JUnit extension reads system properties (`-Dfixture`, `-Dinstaller`) and composes them. The composition is extension-internal; the test sees only an injected `ProxyFixture`. -```java -// CRD-backed: installer puts operator in cluster, fixture deploys via CRDs -new CrdProxyFixture(new ManifestInstaller()) -new CrdProxyFixture(new OlmInstaller()) -new CrdProxyFixture(new MyDownstreamInstaller()) - -// Manifest-backed: deploys proxy directly, no operator -new ManifestProxyFixture() - -// Standalone: local Java process, no Kubernetes -new StandaloneProxyFixture() -``` +Three fixture implementations cover the deployment mechanisms: **`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the operator into the cluster; the fixture applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. @@ -285,13 +274,13 @@ The principle is that **environment differences are absorbed by the fixture, not Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same mechanism as `@Operator` and `@AdmissionWebhook` tags, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. -### Deployment Tests +### Installer Tests -A separate, lightweight test class per supported install method confirms that the installation mechanism produces a working proxy. Each test is a single scenario: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. +The `systemtest-installer` module contains a single smoke test: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. The test does not vary between installers — it is the same test run with different `-Dinstaller` and `-Dfixture` values. CI provides the matrix; the test provides the assertion. -This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the feature test suite; the deployment test only asserts that the installation mechanism puts the proxy in a state where features can run. +This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the `systemtest-feature` module; the installer test only asserts that the installation mechanism puts the proxy in a state where features can run. -Each deployment test has a single reason to fail: the consumer did not see the transformed value. Every possible installation failure collapses into that one observable. No separate assertions per failure mode are needed or wanted; they all manifest identically, and the test name tells you which installation mechanism failed. +Each installer test run has a single reason to fail: the consumer did not see the transformed value. Every possible installation failure collapses into that one observable. No separate assertions per failure mode are needed or wanted; they all manifest identically, and the CI matrix entry tells you which installation mechanism failed. | Install method | Fixture | Installer | File config mechanism | |---|---|---|---| From c3e8f18ae9d9e74f69dea1cc906b93b2e8be61e0 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 15 May 2026 16:55:07 +1200 Subject: [PATCH 06/15] Coherence review: four modules, webhook TCK, fix stale tag language MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Summary now mentions all four test categories and the Installer - "Three categories/modules" → "Four" throughout - Add systemtest-webhook to TCK module list with description - Feature test example: remove namespace parameter, use kafkaClient - Tags section: clarify module boundaries as primary separation - Rejected alternatives: @Operator tag "ensures present", not "selects fixture" - Webhook section: rewrite as two-module story (installer + behaviour) - Affected projects: add systemtest-webhook module - Fix grammar: "An CrdProxyFixture" → "A CrdProxyFixture" Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/xxx-system-test-framework.md | 57 +++++++++++++++----------- 1 file changed, 33 insertions(+), 24 deletions(-) diff --git a/proposals/xxx-system-test-framework.md b/proposals/xxx-system-test-framework.md index fc7b1e01..cf8ba479 100644 --- a/proposals/xxx-system-test-framework.md +++ b/proposals/xxx-system-test-framework.md @@ -2,7 +2,11 @@ ## Summary -Introduce a layered abstraction for system tests that separates test intent (what the proxy should do) from deployment mechanism (how the proxy is stood up). A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. Feature tests become portable across deployment mechanisms — the same test runs against an operator-managed proxy, a manifest-managed proxy, or a downstream distribution — and cheap enough to write before the production code, as a specification. +Introduce a layered abstraction for system tests that separates test intent from deployment mechanism, and organise tests into modules by what they cover: feature behaviour, operator reconciliation, webhook behaviour, and installation validation. + +A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. An `Installer` — the primary downstream extension point — handles getting the operator, CRDs, and RBAC into the cluster independently of how proxies are deployed. + +Feature tests become portable across deployment mechanisms — the same test runs against a CRD-deployed proxy, a manifest-managed proxy, a standalone process, or a downstream distribution — and cheap enough to write before the production code, as a specification. ## Current Situation @@ -33,7 +37,7 @@ Addressing these enables: - **Test-first development**: a developer writing a new filter can write a failing system test as the first commit of their feature branch, without reading framework documentation or asking QE for help. - **Deployment-agnostic feature tests**: the same test runs against an operator-managed proxy, a manifest-managed proxy, or a Helm installation, with no changes to the test body. - **Reliable convergence**: `proxyFixture.apply()` is a blocking call with a defined contract — when it returns, the proxy is serving the requested configuration. Manual polling disappears from test classes. -- **A TCK for downstream distributions**: downstream distributors can implement `ProxyFixture` and run the upstream feature test suite against their distribution without forking the test module. +- **A TCK for downstream distributions**: downstream distributors implement `Installer` for their distribution and run upstream's test modules — feature, operator, installer, and webhook — without forking. ## Proposal @@ -41,13 +45,15 @@ Addressing these enables: The framework needs one organising question answered for every test class: **what is this test covering?** The answer determines which module the test belongs to — not which tag it carries, but which compile-time dependencies it has. -Three categories of system test have fundamentally different concerns: +Four categories of system test have fundamentally different concerns: - **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a correctly-configured proxy exists and is serving traffic. They must not care how the proxy was deployed. They have no Kubernetes dependency. - **Operator tests** — does the operator detect a configuration change and trigger a rolling restart? These tests are explicitly about the operator's reconciliation behaviour. They interact with Kubernetes resources directly and depend on the Kubernetes client. -- **Installer tests** — does a specific installation method (OLM, Helm, kustomize, standalone) produce a working proxy? These tests are about the installation mechanism, not the proxy features. Each test knows its installer. +- **Webhook tests** — does the admission webhook inject sidecars into the right pods? Does it produce a valid pod spec? These tests are about the webhook's API interception behaviour, not proxy functionality. They depend on the Kubernetes client and the webhook being installed. + +- **Installer tests** — does a specific installation method (OLM, Helm, kustomize, standalone) produce a working proxy? These tests are about the installation mechanism, not the proxy features. The test does not vary between installers — CI provides the matrix. These are not tags on a single test suite — they are separate modules with different compile-time dependencies: @@ -55,11 +61,12 @@ These are not tags on a single test suite — they are separate modules with dif |---|---|---|---| | `systemtest-feature` | `ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec` | None | Yes — runs against any fixture | | `systemtest-operator` | Above + `KubernetesCapability`, CRD types, K8s client | Yes | No — requires CRD-based fixture | -| `systemtest-installer` | Above + `KubernetesCapability`, `Installer` | Yes (except standalone) | No — one test per installer | +| `systemtest-webhook` | Above + `KubernetesCapability`, K8s client | Yes | No — requires webhook installed | +| `systemtest-installer` | Above + `KubernetesCapability` | Yes (except standalone) | No — one test per installer | Feature tests do not import Kubernetes types. They cannot accidentally depend on CRDs, namespaces, or client libraries. The module boundary enforces this at compile time, not by convention. -All three modules are consumable as a TCK. A downstream distributor runs feature tests to prove their distribution satisfies the proxy's behavioural contract, installer tests to prove their installation method works, and operator tests to prove operator reconciliation works with their installation. +All four modules are consumable as a TCK. A downstream distributor runs feature tests to prove their distribution satisfies the proxy's behavioural contract, operator tests to prove reconciliation works with their installation, webhook tests to prove admission webhook behaviour, and installer tests to prove their installation method works. ### `ProxyScenario` — Intent Without Deployment @@ -137,12 +144,14 @@ A downstream distributor implements `Installer` for their distribution and compo Tests never instantiate fixtures or installers — the JUnit extension reads system properties (`-Dfixture`, `-Dinstaller`) and composes them. The composition is extension-internal; the test sees only an injected `ProxyFixture`. -Three fixture implementations cover the deployment mechanisms: +Four fixture implementations cover the deployment mechanisms: **`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the operator into the cluster; the fixture applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. **`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. +**`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the operator and admission webhook into the cluster; the fixture creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. + **`StandaloneProxyFixture`**: starts the proxy as a local Java process with a generated configuration file, waits for the port to be ready, and returns a `ProxyHandle` with a localhost bootstrap. No Kubernetes, no installer, no namespaces. `KubernetesCapability` is not available for tests running against this fixture. Kubernetes fixtures use Server-Side Apply. Neither `CrdProxyFixture` nor `ManifestProxyFixture` requires `createOrUpdate` branching or `resourceVersion` management. @@ -174,7 +183,7 @@ interface ProxyHandle { ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible ``` -**Tags as skip conditions**: `@Operator` and `@AdmissionWebhook` are tags that declare a test's infrastructure requirements. If the required component is not present, the extension skips the test before it runs. Tags declare requirements — they do not select fixtures. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). +**Tags as skip conditions**: module boundaries are the primary separation — `systemtest-operator` tests have compile-time Kubernetes dependencies that `systemtest-feature` tests do not. Within Kubernetes-dependent modules, `@Operator` and `@AdmissionWebhook` tags serve as runtime skip conditions: if the required component is not present in the active fixture, the extension skips the test. Tags declare requirements — they do not select fixtures. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). **`KubernetesCapability`**: any test running on Kubernetes can have `KubernetesCapability` injected as a test parameter. This provides general-purpose access to the cluster environment — the namespace the proxy was deployed into and a `KubernetesClient` for observing resource state. Tests running on bare metal do not have `KubernetesCapability` available. @@ -270,7 +279,7 @@ The framework must be runnable across three meaningfully different environments: | Vanilla remote K8s | CI path. OLM optional. LoadBalancer or NodePort ingress. | | OpenShift (OCP) | OLM native. Routes instead of Ingress. Security Context Constraints. | -The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. An `CrdProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. +The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. A `CrdProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same mechanism as `@Operator` and `@AdmissionWebhook` tags, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. @@ -292,15 +301,13 @@ Each installer test run has a single reason to fail: the consumer did not see th | Sidecar injection (webhook) | `SidecarProxyFixture` | `ManifestInstaller` | Kubernetes Secret mount | | Standalone | `StandaloneProxyFixture` | — | File written to local path | -### Admission Webhook Tests - -There are two distinct classes of test for the admission webhook. +### Webhook Tests -**Sidecar injection as a deployment path**: `SidecarProxyFixture` creates a pod with the injection annotation, waits for the webhook to mutate it, waits for the sidecar to be ready, and returns a `ProxyHandle`. Feature tests run against it unchanged. The deployment smoke test for this path is the same file-based filter scenario as every other install method — if the sidecar is injected and the plugin loads, the installation mechanism works. +The webhook touches two modules: -**Webhook behaviour tests**: a separate category that asserts on the Kubernetes API interception layer rather than on proxy behaviour. These tests ask questions that do not produce a `ProxyHandle`: does the webhook inject into pods with annotation X but not Y? Does it produce a valid pod spec? What happens when the webhook is unavailable and `failurePolicy: Ignore`? +**As a deployment path** (`systemtest-installer`): `SidecarProxyFixture` is an installer test entry in the matrix — it proves that sidecar injection produces a working proxy. The same single smoke test runs against it. Feature tests in `systemtest-feature` also run against `SidecarProxyFixture` unchanged. -These are tagged `@AdmissionWebhook` and skip automatically when the webhook is not installed — the same pattern as `@Operator` tests. +**As behaviour under test** (`systemtest-webhook`): a separate module that asserts on the webhook's API interception behaviour. These tests ask questions that do not produce a `ProxyHandle`: does the webhook inject into pods with annotation X but not Y? Does it produce a valid pod spec? What happens when the webhook is unavailable and `failurePolicy: Ignore`? These tests depend on `KubernetesCapability` and require the webhook to be installed. ### TCK Extension Points @@ -314,13 +321,14 @@ mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller Distributors with fundamentally different deployment models (e.g. a custom orchestrator, a managed service) implement `ProxyFixture` directly. Both interfaces have no upstream-specific dependencies in their signatures. -All three test modules are consumable as a TCK: +All four test modules are consumable as a TCK: - **`systemtest-feature`**: downstream proves their distribution satisfies the proxy's behavioural contract — features work regardless of installation method. - **`systemtest-installer`**: downstream proves their installation method produces a working system — their OLM catalog installs correctly, their Helm chart renders valid resources. - **`systemtest-operator`**: downstream proves operator reconciliation works with their installation — change detection, status conditions, rolling restarts all function correctly. +- **`systemtest-webhook`**: downstream proves admission webhook behaviour works with their installation — sidecar injection targets the right pods, produces valid pod specs, and respects failure policies. -Upstream maintains the definition of correct behaviour across all three modules; downstream provides the `Installer` (and optionally the `ProxyFixture`) that adapts the tests to their distribution. +Upstream maintains the definition of correct behaviour across all four modules; downstream provides the `Installer` (and optionally the `ProxyFixture`) that adapts the tests to their distribution. ### What Feature Tests Look Like @@ -359,7 +367,7 @@ private void deployPortIdentifiesNodeWithRecordEncryptionFilter( ```java @Test -void ensureClusterHasEncryptedMessage(String namespace) { +void ensureClusterHasEncryptedMessage() { testKmsFacade.getTestKekManager().generateKek(KEK_PREFIX + topicName); ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() @@ -367,15 +375,15 @@ void ensureClusterHasEncryptedMessage(String namespace) { .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) .build()); - KafkaSteps.createTopic(namespace, topicName, proxy.bootstrap(), 1, 1); - KroxyliciousSteps.produceMessages(namespace, topicName, proxy.bootstrap(), MESSAGE, 1); + kafkaClient.createTopic(topicName, proxy.bootstrap(), 1, 1); + kafkaClient.produceMessages(topicName, proxy.bootstrap(), MESSAGE, 1); - var consumed = KroxyliciousSteps.consumeMessageFromKafkaCluster(...); + var consumed = kafkaClient.consumeMessages(topicName, proxy.bootstrap(), 1); assertThat(consumed).allMatch(r -> !r.getPayload().contains(MESSAGE)); } ``` -The test contains only the Given/When/Then relevant to record encryption. It works against both an operator-managed and a manifest-managed proxy. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. +The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest-feature` test. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. ### What Operator Tests Look Like @@ -433,7 +441,8 @@ void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { **Affected:** - **systemtest-feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec`). No Kubernetes dependency. - **systemtest-operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. -- **systemtest-installer**: new module. One deployment smoke test per supported installation method. +- **systemtest-webhook**: new module. Webhook behaviour tests migrated here. Depends on `KubernetesCapability`. +- **systemtest-installer**: new module. One deployment smoke test, run across the installer/fixture matrix by CI. - **kroxylicious-systemtest (framework)**: the shared framework module providing `ProxyFixture`, `Installer`, `ProxyScenario`, `ProxyHandle`, `KubernetesCapability`, and fixture implementations. - **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest-operator`. @@ -462,7 +471,7 @@ We considered having operator tests use `proxyFixture.apply()` for all mutations We considered exposing an `OperatorCapability` interface to tests, providing methods like `observedGeneration()`, `waitForReconciliation()`, and `currentStatusConditions()`. This would give operator tests a typed API for interacting with the operator's observable state. -On closer examination, operator tests do not need to interact with the operator — they observe its effects on Kubernetes resources. The change detection test reads a checksum annotation; the status condition test reads a resource's status. Both are Kubernetes API observations, not operator interactions. `KubernetesCapability` provides everything these tests need. The operator is infrastructure the extension manages; the `@Operator` tag ensures it is present and selects the right fixture. Making the operator invisible to the test keeps `ProxyHandle` deployment-agnostic and avoids an abstraction that doesn't carry its weight. +On closer examination, operator tests do not need to interact with the operator — they observe its effects on Kubernetes resources. The change detection test reads a checksum annotation; the status condition test reads a resource's status. Both are Kubernetes API observations, not operator interactions. `KubernetesCapability` provides everything these tests need. The operator is infrastructure the extension manages; the `@Operator` tag ensures it is present. Making the operator invisible to the test keeps `ProxyHandle` deployment-agnostic and avoids an abstraction that doesn't carry its weight. ### Merging `KubernetesCapability` into `ProxyHandle` From 6b8f90755f293f7e617791be8b0f0330c7c41c49 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 20 May 2026 16:01:44 +1200 Subject: [PATCH 07/15] Rename proposal to use PR number Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- ...xx-system-test-framework.md => 111-system-test-framework.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{xxx-system-test-framework.md => 111-system-test-framework.md} (99%) diff --git a/proposals/xxx-system-test-framework.md b/proposals/111-system-test-framework.md similarity index 99% rename from proposals/xxx-system-test-framework.md rename to proposals/111-system-test-framework.md index cf8ba479..0fae55f0 100644 --- a/proposals/xxx-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -1,4 +1,4 @@ -# xxx - System Test Framework +# 111 - System Test Framework ## Summary From d13efe2b3b3fdde32615c418305a977bf00f8366 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 2 Jun 2026 14:12:50 +1200 Subject: [PATCH 08/15] Address review feedback: remove RH-specific terminology Replace "delegated to QE" with community-neutral language and frame setup cost as a perceived barrier rather than a statement of fact. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index 0fae55f0..205ad8e1 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -14,7 +14,7 @@ The system test suite covers the right things at the right level. Assertions in Every feature test class has a private `deployXxx()` method that reimplements the same builder/template pattern against the operator's CRD types. Adding a new optional parameter (e.g. `ExperimentalKmsConfig`) requires touching every one of them. Timing workarounds are scattered across test classes with comments pointing at unresolved issues. The convergence question — "is the proxy actually serving the configuration I just applied?" — is answered by ad hoc polling in each test class rather than by a framework-level contract. -This setup cost has a second-order effect: system tests are written after features merge, delegated to QE because they are too expensive for a developer to include in a feature PR. The test framework is the bottleneck, not the assertions. +This setup cost has a second-order effect: system tests are written after features merge, deferred until after merge and often left as an exercise to others because they are perceived as too expensive for a developer to include in a feature PR. The test framework is the bottleneck, not the assertions. The fix required is narrow: a thin setup layer that hides the deployment machinery without changing the assertions at all. From c6a301166534c975063dc13fdd8b249fea5ad3ba Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 2 Jun 2026 14:18:46 +1200 Subject: [PATCH 09/15] Expand TCK acronym at first use Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index 205ad8e1..74722d6f 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -37,7 +37,7 @@ Addressing these enables: - **Test-first development**: a developer writing a new filter can write a failing system test as the first commit of their feature branch, without reading framework documentation or asking QE for help. - **Deployment-agnostic feature tests**: the same test runs against an operator-managed proxy, a manifest-managed proxy, or a Helm installation, with no changes to the test body. - **Reliable convergence**: `proxyFixture.apply()` is a blocking call with a defined contract — when it returns, the proxy is serving the requested configuration. Manual polling disappears from test classes. -- **A TCK for downstream distributions**: downstream distributors implement `Installer` for their distribution and run upstream's test modules — feature, operator, installer, and webhook — without forking. +- **A Technology Compatibility Kit (TCK) for downstream distributions**: downstream distributors implement `Installer` for their distribution and run upstream's test modules — feature, operator, installer, and webhook — without forking. The test modules define what "correct" means; the distributor proves their packaging satisfies it. ## Proposal From 4a195ed5cdd9f063b88ae64a3d1e91ac390510b1 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 3 Jun 2026 16:51:15 +1200 Subject: [PATCH 10/15] Rework KafkaClient section: separate intent, driver, and execution The same separation of concerns that motivates the fixture model applies to how tests interact with Kafka. Three independent axes: test intent (what), client driver (which implementation), and execution environment (where). The KafkaClient interface shows a richer target shape including transactions and consumer group management, enabled by in-process drivers (Java client, librdkafka/Sarama via FFI). CLI drivers implement produce/consume only. Produce and consume are the starting point; richer operations are the target. Assisted-by: Claude claude-opus-4-6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 37 +++++++++++++++++--------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index 74722d6f..cac91e9b 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -239,9 +239,17 @@ If a test in `systemtest-operator` runs but the active fixture is `ManifestProxy ### `KafkaClient` Abstraction -The existing `KafkaClient` interface with its multiple implementations (StrimziTestClient, KcatClient, KafClient, PythonTestClient) — selected at runtime via environment variable — is the right shape and largely works. The gap is off-cluster support. All current implementations run as Kubernetes jobs; an off-cluster client (an embedded Java client in the test JVM, or a client process on a bare metal host) has no namespace and no container image. +The same separation of concerns that motivates the fixture model applies to how tests interact with Kafka. A test expresses intent — produce a message, consume from a topic, commit a transaction — without knowing which client library speaks the protocol or where that client runs. These are three independent axes: -The current interface conflates the core produce/consume contract with Kubernetes-specific machinery: +| Axis | What it answers | Examples | +|---|---|---| +| Test intent | What does the test want to do? | Produce, consume, create topic, transact | +| Client driver | Which protocol implementation? | Java client, librdkafka (via FFI), Sarama (via FFI), kcat CLI | +| Execution environment | Where does the client run? | In-process (test JVM), Kubernetes Job, local process | + +The test sees only intent. The framework composes driver and execution environment. CI can vary them independently — the same feature test runs against the Java client in-process, librdkafka via FFI, or kcat in a Kubernetes pod. + +The existing `KafkaClient` interface (StrimziTestClient, KcatClient, KafClient, PythonTestClient) conflates all three axes. Every implementation is a specific driver running as a Kubernetes Job. The interface itself mixes the produce/consume contract with Kubernetes-specific machinery: ```java KafkaClient inNamespace(String namespace); @@ -249,23 +257,26 @@ String getImage(); void preloadImage(); ``` -These move to a `KubernetesClientCapability`: +The proposed model separates them. `KafkaClient` becomes a richer test-facing API — pure intent, expressed in terms of Kafka operations rather than CLI invocations: ```java interface KafkaClient { - ExecResult produceMessages(...); - List consumeMessages(...); - Optional as(Class capability); -} - -interface KubernetesClientCapability { - KafkaClient inNamespace(String namespace); - String getImage(); - void preloadImage(); + void produce(String topic, String bootstrap, List records); + List consume(String topic, String bootstrap, int count); + void createTopic(String topic, String bootstrap, int partitions, int replicas); + + // Richer operations — enabled by in-process drivers + void produceInTransaction(String topic, String bootstrap, List records); + void commitOffsets(String groupId, String bootstrap, Map offsets); + AdminClient admin(String bootstrap); } ``` -An off-cluster embedded Java client implements `KafkaClient` only. The existing pod-based clients implement both. Code that pre-pulls images or sets a namespace calls `as(KubernetesClientCapability.class)` and skips gracefully if absent. +The current produce/consume contract is sufficient for existing feature tests and is the starting point. The richer operations — transactions, consumer group management, admin — are the target shape. These are difficult or impossible to express via CLI-based drivers (kcat cannot commit a transaction mid-test), but fall out naturally from in-process drivers: the Java client directly, or librdkafka and Sarama via Java Foreign Function Interface. CLI drivers implement produce and consume; in-process drivers implement the full surface. + +The current interface methods that are not test-intent (`inNamespace`, `getImage`, `preloadImage`) are execution-environment concerns and move out of the test-facing contract entirely. The framework wires the right driver and execution environment based on system properties (`-Dclient.driver=java|librdkafka|sarama`, `-Dclient.location=in-process|on-cluster`). + +This separation has a concrete payoff beyond cleanliness: Kroxylicious is a protocol proxy, and proving it works correctly with multiple client implementations — not just the Java client — is a first-class testing concern. A librdkafka or Sarama driver exercised via Java Foreign Function Interface from the test JVM would run in-process (fast, no pod startup cost) while proving protocol compatibility with a fundamentally different client implementation. The same feature test, unchanged, validates behaviour across client libraries. **Bootstrap address pairing**: an off-cluster client needs an externally accessible bootstrap address, not the cluster-internal Service DNS. This pairs naturally with `ProxyHandle.bootstrap(ClientLocation)` — the framework wires client location to bootstrap address at test setup time. The test author calls `proxy.bootstrap()` and receives the right address for the client that is configured. From 7fc352a6819896a7e59a07e8f2828c1b2ba993b2 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 22 Jul 2026 15:28:23 +1200 Subject: [PATCH 11/15] Remove RH-specific 'QE' term from test-first development bullet Assisted-by: Claude Opus 4.6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index cac91e9b..c5a6bf78 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -34,7 +34,7 @@ Three problems prevent this from being the norm: Addressing these enables: -- **Test-first development**: a developer writing a new filter can write a failing system test as the first commit of their feature branch, without reading framework documentation or asking QE for help. +- **Test-first development**: a developer writing a new filter can write a failing system test as the first commit of their feature branch, without reading framework documentation or asking for additional help. - **Deployment-agnostic feature tests**: the same test runs against an operator-managed proxy, a manifest-managed proxy, or a Helm installation, with no changes to the test body. - **Reliable convergence**: `proxyFixture.apply()` is a blocking call with a defined contract — when it returns, the proxy is serving the requested configuration. Manual polling disappears from test classes. - **A Technology Compatibility Kit (TCK) for downstream distributions**: downstream distributors implement `Installer` for their distribution and run upstream's test modules — feature, operator, installer, and webhook — without forking. The test modules define what "correct" means; the distributor proves their packaging satisfies it. From da96d573742c835ac06a0aa683ac1fea55dbf1b0 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 22 Jul 2026 15:56:24 +1200 Subject: [PATCH 12/15] Broaden feature test and Installer definitions for clarity Feature tests care about proxy behaviour under a given configuration, not just happy-path "correctly-configured" scenarios. Installer is the abstraction over installation mechanism (Helm, OLM, manifests) for project components generally (operator, webhook, CRDs, RBAC), not just the operator. Assisted-by: Claude Opus 4.6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index c5a6bf78..0b97cbf6 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -4,7 +4,7 @@ Introduce a layered abstraction for system tests that separates test intent from deployment mechanism, and organise tests into modules by what they cover: feature behaviour, operator reconciliation, webhook behaviour, and installation validation. -A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. An `Installer` — the primary downstream extension point — handles getting the operator, CRDs, and RBAC into the cluster independently of how proxies are deployed. +A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. An `Installer` — the primary downstream extension point — handles getting project components (operator, webhook, CRDs, RBAC) into the cluster independently of how proxies are deployed. Feature tests become portable across deployment mechanisms — the same test runs against a CRD-deployed proxy, a manifest-managed proxy, a standalone process, or a downstream distribution — and cheap enough to write before the production code, as a specification. @@ -47,7 +47,7 @@ The framework needs one organising question answered for every test class: **wha Four categories of system test have fundamentally different concerns: -- **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a correctly-configured proxy exists and is serving traffic. They must not care how the proxy was deployed. They have no Kubernetes dependency. +- **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a proxy has been started with a given configuration. They must not care how the proxy was deployed. They have no Kubernetes dependency. - **Operator tests** — does the operator detect a configuration change and trigger a rolling restart? These tests are explicitly about the operator's reconciliation behaviour. They interact with Kubernetes resources directly and depend on the Kubernetes client. @@ -117,11 +117,11 @@ interface ProxyFixture { This is an explicit call — not magic JUnit injection — because the test author needs to understand that `apply()` is a blocking operation that includes convergence waiting. Hiding it behind injection would obscure the framework's most important contract. -Fixture implementations span two independent concerns: how the infrastructure is installed (CRDs, RBAC, operator Deployment) and how proxy instances are deployed. These concerns are separated by composing a `ProxyFixture` with an `Installer`. +Fixture implementations span two independent concerns: how the infrastructure is installed (operator, webhook, CRDs, RBAC) and how proxy instances are deployed. These concerns are separated by composing a `ProxyFixture` with an `Installer`. ### `Installer` — Infrastructure Installation -`Installer` handles getting the operator, CRDs, RBAC rules, and ServiceAccounts into the cluster. It is a **public interface** — the primary extension point for downstream distributors, who typically vary only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed. +`Installer` handles getting project components — operator, webhook, CRDs, RBAC rules, ServiceAccounts — into the cluster. It is a **public interface** — the primary extension point for downstream distributors, who typically vary only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed. ```java interface Installer { @@ -150,7 +150,7 @@ Four fixture implementations cover the deployment mechanisms: **`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. -**`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the operator and admission webhook into the cluster; the fixture creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. +**`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the admission webhook into the cluster; the fixture creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. **`StandaloneProxyFixture`**: starts the proxy as a local Java process with a generated configuration file, waits for the port to be ready, and returns a `ProxyHandle` with a localhost bootstrap. No Kubernetes, no installer, no namespaces. `KubernetesCapability` is not available for tests running against this fixture. From 1b5ff8a370dabac5d1bd22453aebf091e6d793af Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 23 Jul 2026 13:39:59 +1200 Subject: [PATCH 13/15] Introduce OperatorFixture, WebhookFixture, and component-based Installer Reshape the fixture model so each test module has a fixture type matched to what it exercises: ProxyFixture for feature tests, OperatorFixture for operator reconciliation tests, WebhookFixture for admission webhook tests. Installer gains a Component enum so fixtures request only what they need (OPERATOR, WEBHOOK). Move test modules under a systemtest/ parent for physical grouping. Extract the Installer interface into a public API module for downstream extensibility. Specific installer implementations (Helm, OLM, manifests) are recognised as likely but out of scope. Tags shift from component requirements to runtime environment skip conditions (e.g. skip when no Kubernetes cluster is available). Assisted-by: Claude Opus 4.6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 130 ++++++++++++------------- 1 file changed, 62 insertions(+), 68 deletions(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index 0b97cbf6..fea573f2 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -30,7 +30,7 @@ Three problems prevent this from being the norm: 2. **No convergence contract**: the framework does not define when the proxy is ready. Each test class independently polls for readiness, with varying strategies and varying reliability. -3. **Operator coupling**: every test implicitly requires the operator. Feature tests — which care only that a correctly-configured proxy is serving traffic — cannot run without the full operator installation. This conflates feature correctness with operator correctness and prevents fast local iteration. +3. **Operator coupling**: every test implicitly requires the operator. Feature tests — which care only that a proxy has been started with a given configuration — cannot run without the full operator installation. This conflates feature correctness with operator correctness and prevents fast local iteration. Addressing these enables: @@ -59,10 +59,10 @@ These are not tags on a single test suite — they are separate modules with dif | Module | Depends on | Kubernetes dependency | Portable across fixtures | |---|---|---|---| -| `systemtest-feature` | `ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec` | None | Yes — runs against any fixture | -| `systemtest-operator` | Above + `KubernetesCapability`, CRD types, K8s client | Yes | No — requires CRD-based fixture | -| `systemtest-webhook` | Above + `KubernetesCapability`, K8s client | Yes | No — requires webhook installed | -| `systemtest-installer` | Above + `KubernetesCapability` | Yes (except standalone) | No — one test per installer | +| `systemtest/feature` | `ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec` | None | Yes — runs against any `ProxyFixture` | +| `systemtest/operator` | `OperatorFixture`, `KubernetesCapability`, CRD types, K8s client | Yes | No — requires operator installed | +| `systemtest/webhook` | `WebhookFixture`, `KubernetesCapability`, K8s client | Yes | No — requires webhook installed | +| `systemtest/installer` | `ProxyFixture`, `KubernetesCapability` | Yes (except standalone) | No — one test per installer | Feature tests do not import Kubernetes types. They cannot accidentally depend on CRDs, namespaces, or client libraries. The module boundary enforces this at compile time, not by convention. @@ -117,40 +117,34 @@ interface ProxyFixture { This is an explicit call — not magic JUnit injection — because the test author needs to understand that `apply()` is a blocking operation that includes convergence waiting. Hiding it behind injection would obscure the framework's most important contract. -Fixture implementations span two independent concerns: how the infrastructure is installed (operator, webhook, CRDs, RBAC) and how proxy instances are deployed. These concerns are separated by composing a `ProxyFixture` with an `Installer`. +Test fixtures span two independent concerns: how infrastructure components are installed (operator, webhook, CRDs, RBAC) and what the test is exercising. The `Installer` handles the first concern; the fixture type handles the second. Feature tests use `ProxyFixture`, operator tests use `OperatorFixture`, and webhook tests use `WebhookFixture`. ### `Installer` — Infrastructure Installation -`Installer` handles getting project components — operator, webhook, CRDs, RBAC rules, ServiceAccounts — into the cluster. It is a **public interface** — the primary extension point for downstream distributors, who typically vary only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed. +`Installer` handles getting project components — operator, webhook, CRDs, RBAC rules, ServiceAccounts — into the cluster. It is a **public interface** — the primary extension point for downstream distributors, who typically vary only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed. The `Installer` interface and supporting types live in a dedicated API module so that downstream distributors can depend on it without pulling in framework internals. ```java +enum Component { OPERATOR, WEBHOOK } + interface Installer { - void install(); - void uninstall(); + void install(Component component); + void uninstall(Component component); } ``` -Upstream ships implementations for each supported installation method: - -| Installer | What it installs | -|---|---| -| `ManifestInstaller` | Operator via kustomize/raw manifests (upstream default) | -| `HelmInstaller` | Operator via Helm chart | -| `OlmInstaller` | Operator via OLM catalog | +We recognise there is likely to be a set of standard installers which need customising only by simple parameters such as a Helm chart URI or an OLM bundle image. Specifying those implementations and their configuration surface is out of scope for this proposal. A downstream distributor could configure an upstream-provided implementation or implement the interface directly. -A downstream distributor implements `Installer` for their distribution and composes it with upstream's fixture — no need to reimplement proxy deployment or convergence logic. +### `ProxyFixture` Implementations -### Fixture Implementations +Tests never instantiate fixtures or installers — the JUnit extension reads system properties (`-Dfixture`, `-Dinstaller`) and composes them. The composition is extension-internal; the test sees only an injected fixture. -Tests never instantiate fixtures or installers — the JUnit extension reads system properties (`-Dfixture`, `-Dinstaller`) and composes them. The composition is extension-internal; the test sees only an injected `ProxyFixture`. +Four `ProxyFixture` implementations cover the deployment mechanisms. Feature tests run against all of them unchanged — the test calls `apply(scenario)` and gets a `ProxyHandle` regardless of how the proxy was deployed. -Four fixture implementations cover the deployment mechanisms: - -**`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the operator into the cluster; the fixture applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. +**`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. Calls `install(OPERATOR)` to put the operator into the cluster, then applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. **`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. -**`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. The installer puts the admission webhook into the cluster; the fixture creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. +**`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. Calls `install(WEBHOOK)` to put the admission webhook into the cluster, then creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. **`StandaloneProxyFixture`**: starts the proxy as a local Java process with a generated configuration file, waits for the port to be ready, and returns a `ProxyHandle` with a localhost bootstrap. No Kubernetes, no installer, no namespaces. `KubernetesCapability` is not available for tests running against this fixture. @@ -158,6 +152,18 @@ Kubernetes fixtures use Server-Side Apply. Neither `CrdProxyFixture` nor `Manife **A note on convergence**: the framework waits for the best observable signal, not a guarantee. There is an inherent gap between "the operator updated the Deployment" and "the new pods are handling traffic." The `ProxyFixture` contract is: when `apply()` returns, the proxy is serving the requested configuration to the best observable precision. +### `OperatorFixture` — Reconciliation Testing + +Operator tests are explicitly about the operator's behaviour — they need to manipulate CRDs and observe how the operator reconciles them. `OperatorFixture` calls `install(OPERATOR)` internally and exposes the operator's surface directly: applying CRD changes, observing reconciliation outcomes, and verifying that the operator reacts correctly to configuration changes, invalid inputs, and resource deletion. + +Tests in `systemtest/operator` receive an `OperatorFixture` rather than a `ProxyFixture`. The test drives the operator through its CRD API and asserts the resulting cluster state. + +### `WebhookFixture` — Admission Webhook Testing + +Webhook tests are about the admission webhook's mutation behaviour — they need to create pods and verify that the webhook mutates them correctly. `WebhookFixture` calls `install(WEBHOOK)` internally and exposes the webhook's surface: creating annotated pods, inspecting the resulting pod spec, and verifying that injection produces valid sidecar configurations. + +Tests in `systemtest/webhook` receive a `WebhookFixture` rather than a `ProxyFixture`. The test drives the webhook through pod creation and asserts the mutation outcome. + ### `ProxyHandle` — A Token of Convergence The only way to obtain a `ProxyHandle` is through `ProxyFixture.apply()`. This means a test cannot accidentally interact with the proxy before convergence has been waited for. @@ -177,19 +183,21 @@ interface ProxyHandle { ### Injection Model and Tags -`ProxyFixture` is injected by the JUnit extension at class scope — it is an environment configuration concern, long-lived, with no timing implications. `ProxyHandle` is always obtained explicitly by calling `proxyFixture.apply()` in the test body. This call is blocking and includes convergence waiting; making it explicit ensures the test author understands the contract. +The appropriate fixture (`ProxyFixture`, `OperatorFixture`, or `WebhookFixture`) is injected by the JUnit extension at class scope — it is an environment configuration concern, long-lived, with no timing implications. For feature tests, `ProxyHandle` is always obtained explicitly by calling `proxyFixture.apply()` in the test body. This call is blocking and includes convergence waiting; making it explicit ensures the test author understands the contract. ```java ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible ``` -**Tags as skip conditions**: module boundaries are the primary separation — `systemtest-operator` tests have compile-time Kubernetes dependencies that `systemtest-feature` tests do not. Within Kubernetes-dependent modules, `@Operator` and `@AdmissionWebhook` tags serve as runtime skip conditions: if the required component is not present in the active fixture, the extension skips the test. Tags declare requirements — they do not select fixtures. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). +**Module boundaries as compile-time separation**: `systemtest/feature` tests depend only on `ProxyFixture` and have no Kubernetes dependency. `systemtest/operator` tests depend on `OperatorFixture` and the Kubernetes client. `systemtest/webhook` tests depend on `WebhookFixture` and the Kubernetes client. These are compile-time boundaries that control what a test *can* depend on. + +**Tags as runtime skip conditions**: module boundaries cannot account for runtime environment availability. Tags like `@Kubernetes` declare that a test requires a specific runtime environment and cause the extension to skip the test when that environment is not available — e.g. skipping `systemtest/operator` tests when no cluster is reachable. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). **`KubernetesCapability`**: any test running on Kubernetes can have `KubernetesCapability` injected as a test parameter. This provides general-purpose access to the cluster environment — the namespace the proxy was deployed into and a `KubernetesClient` for observing resource state. Tests running on bare metal do not have `KubernetesCapability` available. | Concept | How obtained | Reason | |---|---|---| -| `ProxyFixture` | Injected (class-scoped) | Environment config, no timing implications | +| `ProxyFixture` / `OperatorFixture` / `WebhookFixture` | Injected (class-scoped) | Environment config, no timing implications | | `KubernetesCapability` | Injected for Kubernetes-deployed tests | Namespace and client access for resource observation | | `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | @@ -206,11 +214,6 @@ interface KubernetesCapability { The namespace is managed by the fixture. Each `apply()` call deploys into a namespace the fixture controls; the test discovers it through the capability rather than supplying it. This keeps `ProxyScenario` free of deployment concerns while giving tests the access they need for resource observation and client operations. -### Tags and Component Requirements - -`@Operator` and `@AdmissionWebhook` are skip tags — they declare that a test requires a specific component and cause the extension to skip the test if that component is not present. The operator itself is infrastructure: the extension uses the configured `Installer` to deploy it before operator-tagged tests run. - -The test does not interact with the operator directly. It interacts with the proxy (via `ProxyHandle`) and with Kubernetes resources (via `KubernetesCapability`). The operator is the mechanism that makes the proxy appear in response to CRDs; the test observes the result, not the mechanism. ### Fixture and Installer Selection @@ -233,9 +236,7 @@ mvn test -Dfixture=manifest mvn test -Dfixture=standalone ``` -The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `-Dinstaller` is not specified, the fixture uses its default (`ManifestInstaller` for operator fixtures). Standalone and manifest fixtures do not take an installer. - -If a test in `systemtest-operator` runs but the active fixture is `ManifestProxyFixture` or `StandaloneProxyFixture`, the test skips — the operator is not present, and the fixture cannot satisfy the requirement. +The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `-Dinstaller` is not specified, the fixture uses its default (manifest-based for fixtures that require an installer). Standalone and manifest fixtures do not take an installer. ### `KafkaClient` Abstraction @@ -292,54 +293,53 @@ The framework must be runnable across three meaningfully different environments: The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. A `CrdProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. -Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same mechanism as `@Operator` and `@AdmissionWebhook` tags, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. +Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same skip mechanism, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. ### Installer Tests -The `systemtest-installer` module contains a single smoke test: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. The test does not vary between installers — it is the same test run with different `-Dinstaller` and `-Dfixture` values. CI provides the matrix; the test provides the assertion. +The `systemtest/installer` module contains a single smoke test: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. The test does not vary between installers — it is the same test run with different `-Dinstaller` and `-Dfixture` values. CI provides the matrix; the test provides the assertion. -This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the `systemtest-feature` module; the installer test only asserts that the installation mechanism puts the proxy in a state where features can run. +This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the `systemtest/feature` module; the installer test only asserts that the installation mechanism puts the proxy in a state where features can run. Each installer test run has a single reason to fail: the consumer did not see the transformed value. Every possible installation failure collapses into that one observable. No separate assertions per failure mode are needed or wanted; they all manifest identically, and the CI matrix entry tells you which installation mechanism failed. | Install method | Fixture | Installer | File config mechanism | |---|---|---|---| -| CRD (manifests) | `CrdProxyFixture` | `ManifestInstaller` | Kubernetes Secret mount | -| CRD (Helm) | `CrdProxyFixture` | `HelmInstaller` | Kubernetes Secret mount | -| CRD (OLM) | `CrdProxyFixture` | `OlmInstaller` | Kubernetes Secret mount | -| Manifest (Helm, no operator) | `ManifestProxyFixture` | — | Kubernetes Secret mount | -| Manifest (Kustomize / raw YAML) | `ManifestProxyFixture` | — | Kubernetes Secret mount | -| Sidecar injection (webhook) | `SidecarProxyFixture` | `ManifestInstaller` | Kubernetes Secret mount | +| CRD (manifests) | `CrdProxyFixture` | manifest-based | Kubernetes Secret mount | +| CRD (Helm) | `CrdProxyFixture` | Helm-based | Kubernetes Secret mount | +| CRD (OLM) | `CrdProxyFixture` | OLM-based | Kubernetes Secret mount | +| Manifest (no operator) | `ManifestProxyFixture` | — | Kubernetes Secret mount | +| Sidecar injection (webhook) | `SidecarProxyFixture` | manifest-based | Kubernetes Secret mount | | Standalone | `StandaloneProxyFixture` | — | File written to local path | ### Webhook Tests The webhook touches two modules: -**As a deployment path** (`systemtest-installer`): `SidecarProxyFixture` is an installer test entry in the matrix — it proves that sidecar injection produces a working proxy. The same single smoke test runs against it. Feature tests in `systemtest-feature` also run against `SidecarProxyFixture` unchanged. +**As a deployment path** (`systemtest/installer`): `SidecarProxyFixture` is an installer test entry in the matrix — it proves that sidecar injection produces a working proxy. The same single smoke test runs against it. Feature tests in `systemtest/feature` also run against `SidecarProxyFixture` unchanged. -**As behaviour under test** (`systemtest-webhook`): a separate module that asserts on the webhook's API interception behaviour. These tests ask questions that do not produce a `ProxyHandle`: does the webhook inject into pods with annotation X but not Y? Does it produce a valid pod spec? What happens when the webhook is unavailable and `failurePolicy: Ignore`? These tests depend on `KubernetesCapability` and require the webhook to be installed. +**As behaviour under test** (`systemtest/webhook`): a separate module that uses `WebhookFixture` to assert on the webhook's API interception behaviour. These tests ask questions that do not produce a `ProxyHandle`: does the webhook inject into pods with annotation X but not Y? Does it produce a valid pod spec? What happens when the webhook is unavailable and `failurePolicy: Ignore`? ### TCK Extension Points -The framework provides two public interfaces for downstream extensibility: `ProxyFixture` and `Installer`. +The framework provides public interfaces for downstream extensibility: `Installer`, `ProxyFixture`, `OperatorFixture`, and `WebhookFixture`. -Most downstream distributors differ only in how the operator is installed — their own OLM catalog, their own Helm chart, a different RBAC configuration. These distributors implement `Installer` and compose it with upstream's `CrdProxyFixture`, inheriting all proxy deployment and convergence logic: +Most downstream distributors differ only in how components are installed — their own OLM catalog, their own Helm chart, a different RBAC configuration. These distributors implement `Installer` (or configure an upstream-provided one) and compose it with upstream's fixtures, inheriting all deployment and convergence logic: ```bash mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller ``` -Distributors with fundamentally different deployment models (e.g. a custom orchestrator, a managed service) implement `ProxyFixture` directly. Both interfaces have no upstream-specific dependencies in their signatures. +Distributors with fundamentally different deployment models (e.g. a custom orchestrator, a managed service) may also need custom fixture implementations. The `Installer` and fixture interfaces have no upstream-specific dependencies in their signatures. All four test modules are consumable as a TCK: -- **`systemtest-feature`**: downstream proves their distribution satisfies the proxy's behavioural contract — features work regardless of installation method. -- **`systemtest-installer`**: downstream proves their installation method produces a working system — their OLM catalog installs correctly, their Helm chart renders valid resources. -- **`systemtest-operator`**: downstream proves operator reconciliation works with their installation — change detection, status conditions, rolling restarts all function correctly. -- **`systemtest-webhook`**: downstream proves admission webhook behaviour works with their installation — sidecar injection targets the right pods, produces valid pod specs, and respects failure policies. +- **`systemtest/feature`**: downstream proves their distribution satisfies the proxy's behavioural contract — features work regardless of installation method. +- **`systemtest/installer`**: downstream proves their installation method produces a working system — their OLM catalog installs correctly, their Helm chart renders valid resources. +- **`systemtest/operator`**: downstream proves operator reconciliation works with their installation — change detection, status conditions, rolling restarts all function correctly. +- **`systemtest/webhook`**: downstream proves admission webhook behaviour works with their installation — sidecar injection targets the right pods, produces valid pod specs, and respects failure policies. -Upstream maintains the definition of correct behaviour across all four modules; downstream provides the `Installer` (and optionally the `ProxyFixture`) that adapts the tests to their distribution. +Upstream maintains the definition of correct behaviour across all four modules. Most downstream distributors need only implement `Installer` for their installation mechanism. Distributors with fundamentally different deployment models may also need custom fixture implementations. ### What Feature Tests Look Like @@ -394,7 +394,7 @@ void ensureClusterHasEncryptedMessage() { } ``` -The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest-feature` test. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. +The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest/feature` test. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. ### What Operator Tests Look Like @@ -421,10 +421,9 @@ void shouldUpdateWhenFilterConfigurationChanges(String namespace) { **After**: ```java -@Operator @Test void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { - ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() + ProxyHandle proxy = operatorFixture.apply(ProxyScenario.builder() .withUpstream(clusterName) .withFilter(new SimpleTransformFilterSpec("foo", "bar")) .build()); @@ -445,17 +444,17 @@ void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { } ``` -`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `readChecksumAnnotation()` is called against stable state — no polling required to establish the baseline. This test lives in the `systemtest-operator` module, which has compile-time access to `KubernetesCapability` and the CRD types. The test does not interact with the operator directly — it observes the operator's effect on Kubernetes resources. The resource mutation is intentionally direct — this test exists to prove the operator detects and responds to changes made outside the fixture. +`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `readChecksumAnnotation()` is called against stable state — no polling required to establish the baseline. This test lives in the `systemtest/operator` module, which uses `OperatorFixture` and has compile-time access to `KubernetesCapability` and the CRD types. The resource mutation is intentionally direct — this test exists to prove the operator detects and responds to changes made outside the fixture. ## Affected/Not Affected Projects **Affected:** -- **systemtest-feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec`). No Kubernetes dependency. -- **systemtest-operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. -- **systemtest-webhook**: new module. Webhook behaviour tests migrated here. Depends on `KubernetesCapability`. -- **systemtest-installer**: new module. One deployment smoke test, run across the installer/fixture matrix by CI. -- **kroxylicious-systemtest (framework)**: the shared framework module providing `ProxyFixture`, `Installer`, `ProxyScenario`, `ProxyHandle`, `KubernetesCapability`, and fixture implementations. -- **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest-operator`. +- **systemtest/feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec`). No Kubernetes dependency. +- **systemtest/operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. +- **systemtest/webhook**: new module. Webhook behaviour tests migrated here. Depends on `KubernetesCapability`. +- **systemtest/installer**: new module. One deployment smoke test, run across the installer/fixture matrix by CI. +- **systemtest (parent)**: parent module with child test modules (`feature`, `operator`, `webhook`, `installer`). Contains the shared framework (fixture interfaces, `ProxyScenario`, `ProxyHandle`, `KubernetesCapability`, fixture implementations) and a public API module for the `Installer` interface and supporting types. +- **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest/operator`. **Not affected:** - **kroxylicious-proxy (runtime)**: no production code changes. The framework abstracts over the proxy; it does not change it. @@ -478,11 +477,6 @@ We considered having the JUnit extension inject `ProxyHandle` directly as a test We considered having operator tests use `proxyFixture.apply()` for all mutations, including the mid-test configuration changes that operator tests need to assert on. However, operator tests exist specifically to prove that the operator detects mutations made outside the fixture — bypassing the fixture for the mid-test mutation is the point of the test. Routing those mutations through the fixture would test the fixture's update path, not the operator's reconciliation behaviour. -### `OperatorCapability` as a test-facing API - -We considered exposing an `OperatorCapability` interface to tests, providing methods like `observedGeneration()`, `waitForReconciliation()`, and `currentStatusConditions()`. This would give operator tests a typed API for interacting with the operator's observable state. - -On closer examination, operator tests do not need to interact with the operator — they observe its effects on Kubernetes resources. The change detection test reads a checksum annotation; the status condition test reads a resource's status. Both are Kubernetes API observations, not operator interactions. `KubernetesCapability` provides everything these tests need. The operator is infrastructure the extension manages; the `@Operator` tag ensures it is present. Making the operator invisible to the test keeps `ProxyHandle` deployment-agnostic and avoids an abstraction that doesn't carry its weight. ### Merging `KubernetesCapability` into `ProxyHandle` From 5fa10c1ef977384012f95da4327b4fa897aab36d Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Fri, 24 Jul 2026 09:04:27 +1200 Subject: [PATCH 14/15] Introduce KafkaClusterFixture for pluggable upstream cluster provisioning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream Kafka provisioning was an ambient assumption — clusterName appeared in examples without explaining where it comes from. KafkaClusterFixture and KafkaClusterHandle follow the same handle-based convergence pattern as ProxyFixture/ProxyHandle, making cluster provisioning explicit and pluggable across implementations (in-VM, TestContainers, Strimzi, managed services). Assisted-by: Claude Opus 4.6 Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 79 ++++++++++++++++++-------- 1 file changed, 55 insertions(+), 24 deletions(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index fea573f2..2d411ff6 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -4,7 +4,7 @@ Introduce a layered abstraction for system tests that separates test intent from deployment mechanism, and organise tests into modules by what they cover: feature behaviour, operator reconciliation, webhook behaviour, and installation validation. -A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. An `Installer` — the primary downstream extension point — handles getting project components (operator, webhook, CRDs, RBAC) into the cluster independently of how proxies are deployed. +A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. A `KafkaClusterFixture` provisions the upstream Kafka cluster and returns a `KafkaClusterHandle` — the same handle-based convergence pattern, making cluster provisioning pluggable across implementations (TestContainers, Strimzi, in-VM). An `Installer` — the primary downstream extension point — handles getting project components (operator, webhook, CRDs, RBAC) into the cluster independently of how proxies are deployed. Feature tests become portable across deployment mechanisms — the same test runs against a CRD-deployed proxy, a manifest-managed proxy, a standalone process, or a downstream distribution — and cheap enough to write before the production code, as a specification. @@ -59,10 +59,10 @@ These are not tags on a single test suite — they are separate modules with dif | Module | Depends on | Kubernetes dependency | Portable across fixtures | |---|---|---|---| -| `systemtest/feature` | `ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec` | None | Yes — runs against any `ProxyFixture` | -| `systemtest/operator` | `OperatorFixture`, `KubernetesCapability`, CRD types, K8s client | Yes | No — requires operator installed | +| `systemtest/feature` | `ProxyFixture`, `KafkaClusterFixture`, `ProxyScenario`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec` | None | Yes — runs against any `ProxyFixture` | +| `systemtest/operator` | `OperatorFixture`, `KafkaClusterFixture`, `KubernetesCapability`, CRD types, K8s client | Yes | No — requires operator installed | | `systemtest/webhook` | `WebhookFixture`, `KubernetesCapability`, K8s client | Yes | No — requires webhook installed | -| `systemtest/installer` | `ProxyFixture`, `KubernetesCapability` | Yes (except standalone) | No — one test per installer | +| `systemtest/installer` | `ProxyFixture`, `KafkaClusterFixture`, `KubernetesCapability` | Yes (except standalone) | No — one test per installer | Feature tests do not import Kubernetes types. They cannot accidentally depend on CRDs, namespaces, or client libraries. The module boundary enforces this at compile time, not by convention. @@ -74,18 +74,20 @@ A plain Java value object describing what configuration the proxy should have. N ```java ProxyScenario scenario = ProxyScenario.builder() - .withUpstream(clusterName) + .withUpstream(upstream) .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) .build(); ProxyScenario scenario = ProxyScenario.builder() - .withUpstream(clusterName) + .withUpstream(upstream) .withFilter(new RecordEncryptionFilterSpec(testKmsFacade) .withExperimentalConfig(config)) .withDownstreamTls(tls) .build(); ``` +Where `upstream` is a `KafkaClusterHandle` obtained from `kafkaClusterFixture.provision()` — see [KafkaClusterFixture](#kafkaclusterfixture--upstream-cluster-provisioning). + ### `FilterSpec` — The Filter DSL `FilterSpec` is an interface for expressing which filter the proxy should run and how it should be configured — in terms of the filter's purpose, not its deployment mechanics. No template classes, no filter type names, no namespaces visible to the test author. @@ -181,15 +183,40 @@ interface ProxyHandle { `waitForRestart()` is on `ProxyHandle` rather than on any capability because restarting the proxy is meaningful across all fixture types — on Kubernetes the fixture observes the Deployment rollout; on bare metal the fixture manages the process restart directly. The concept is universal; the mechanism is fixture-specific. +### `KafkaClusterFixture` — Upstream Cluster Provisioning + +Tests need an upstream Kafka cluster to proxy. Provisioning that cluster is a pluggable concern with meaningfully different implementations: an in-VM broker for fast local iteration, TestContainers for a real broker without a Kubernetes dependency, Strimzi for Kubernetes-native clusters, or a managed service. + +`KafkaClusterFixture` follows the same pattern as the other fixtures: the extension injects it, the test author calls `provision()` explicitly, and the returned `KafkaClusterHandle` is a token proving the cluster is ready. + +```java +interface KafkaClusterFixture { + KafkaClusterHandle provision(); +} + +interface KafkaClusterHandle { + String bootstrap(); +} +``` + +`KafkaClusterHandle` serves the same role as `ProxyHandle` — you cannot interact with the cluster before provisioning has completed. `ProxyScenario.withUpstream()` takes a `KafkaClusterHandle` rather than a raw string, making the dependency on a provisioned cluster explicit in the type system. + +**Lifecycle is the test author's choice.** A `KafkaClusterHandle` provisioned in `@BeforeAll` is shared across all tests in the class — sensible when tests don't mutate cluster state and the cluster is expensive to provision. A handle provisioned in the test body is per-test. The extension manages cleanup at the appropriate scope. This is the same model as `ProxyFixture` — the framework provides the mechanism; the test author decides the scope. + +**Selection**: the `KafkaClusterFixture` implementation is selected via system property (`-Dkafka.cluster=testcontainers|strimzi|invm`), consistent with fixture and installer selection. + +Specifying the full set of `KafkaClusterFixture` implementations and their configuration surface is out of scope for this proposal. The key contribution is establishing that upstream cluster provisioning is a pluggable, fixture-managed concern with a handle-based convergence contract — not an ambient assumption. + ### Injection Model and Tags -The appropriate fixture (`ProxyFixture`, `OperatorFixture`, or `WebhookFixture`) is injected by the JUnit extension at class scope — it is an environment configuration concern, long-lived, with no timing implications. For feature tests, `ProxyHandle` is always obtained explicitly by calling `proxyFixture.apply()` in the test body. This call is blocking and includes convergence waiting; making it explicit ensures the test author understands the contract. +The appropriate fixtures (`ProxyFixture`, `OperatorFixture`, `WebhookFixture`, and `KafkaClusterFixture`) are injected by the JUnit extension at class scope — they are environment configuration concerns, long-lived, with no timing implications. `ProxyHandle` and `KafkaClusterHandle` are always obtained explicitly by calling `apply()` or `provision()` in the test body. These calls are blocking and include convergence waiting; making them explicit ensures the test author understands the contract. ```java -ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible +KafkaClusterHandle upstream = kafkaClusterFixture.provision(); // explicit — provisioning visible +ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible ``` -**Module boundaries as compile-time separation**: `systemtest/feature` tests depend only on `ProxyFixture` and have no Kubernetes dependency. `systemtest/operator` tests depend on `OperatorFixture` and the Kubernetes client. `systemtest/webhook` tests depend on `WebhookFixture` and the Kubernetes client. These are compile-time boundaries that control what a test *can* depend on. +**Module boundaries as compile-time separation**: `systemtest/feature` tests depend only on `ProxyFixture`, `KafkaClusterFixture`, and their handle types, and have no Kubernetes dependency. `systemtest/operator` tests depend on `OperatorFixture` and the Kubernetes client. `systemtest/webhook` tests depend on `WebhookFixture` and the Kubernetes client. These are compile-time boundaries that control what a test *can* depend on. **Tags as runtime skip conditions**: module boundaries cannot account for runtime environment availability. Tags like `@Kubernetes` declare that a test requires a specific runtime environment and cause the extension to skip the test when that environment is not available — e.g. skipping `systemtest/operator` tests when no cluster is reachable. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). @@ -198,7 +225,9 @@ ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence v | Concept | How obtained | Reason | |---|---|---| | `ProxyFixture` / `OperatorFixture` / `WebhookFixture` | Injected (class-scoped) | Environment config, no timing implications | +| `KafkaClusterFixture` | Injected (class-scoped) | Environment config, no timing implications | | `KubernetesCapability` | Injected for Kubernetes-deployed tests | Namespace and client access for resource observation | +| `KafkaClusterHandle` | Always explicit via `provision()` | Provisioning is a blocking operation; must be visible | | `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | ### `KubernetesCapability` — Cluster Environment Access @@ -217,14 +246,14 @@ The namespace is managed by the fixture. Each `apply()` call deploys into a name ### Fixture and Installer Selection -Fixture and installer are selected independently via system properties: +Fixture, installer, and Kafka cluster provisioning are selected independently via system properties: ```bash -# Default upstream: operator installed via manifests, proxy deployed via CRDs -mvn test -Dfixture=crd -Dinstaller=manifest +# Default upstream: operator installed via manifests, proxy deployed via CRDs, TestContainers Kafka +mvn test -Dfixture=crd -Dinstaller=manifest -Dkafka.cluster=testcontainers -# OLM installation -mvn test -Dfixture=crd -Dinstaller=olm +# OLM installation with Strimzi-managed Kafka +mvn test -Dfixture=crd -Dinstaller=olm -Dkafka.cluster=strimzi # Downstream custom installer, upstream fixture mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller @@ -232,11 +261,11 @@ mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller # Manifest-managed proxy (no operator) mvn test -Dfixture=manifest -# Standalone -mvn test -Dfixture=standalone +# Standalone with in-VM Kafka (fast local iteration) +mvn test -Dfixture=standalone -Dkafka.cluster=invm ``` -The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `-Dinstaller` is not specified, the fixture uses its default (manifest-based for fixtures that require an installer). Standalone and manifest fixtures do not take an installer. +The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `-Dinstaller` is not specified, the fixture uses its default (manifest-based for fixtures that require an installer). Standalone and manifest fixtures do not take an installer. When `-Dkafka.cluster` is not specified, the extension uses a sensible default for the fixture type. ### `KafkaClient` Abstraction @@ -322,7 +351,7 @@ The webhook touches two modules: ### TCK Extension Points -The framework provides public interfaces for downstream extensibility: `Installer`, `ProxyFixture`, `OperatorFixture`, and `WebhookFixture`. +The framework provides public interfaces for downstream extensibility: `Installer`, `ProxyFixture`, `OperatorFixture`, `WebhookFixture`, and `KafkaClusterFixture`. Most downstream distributors differ only in how components are installed — their own OLM catalog, their own Helm chart, a different RBAC configuration. These distributors implement `Installer` (or configure an upstream-provided one) and compose it with upstream's fixtures, inheriting all deployment and convergence logic: @@ -381,8 +410,9 @@ private void deployPortIdentifiesNodeWithRecordEncryptionFilter( void ensureClusterHasEncryptedMessage() { testKmsFacade.getTestKekManager().generateKek(KEK_PREFIX + topicName); + KafkaClusterHandle upstream = kafkaClusterFixture.provision(); ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() - .withUpstream(clusterName) + .withUpstream(upstream) .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) .build()); @@ -394,7 +424,7 @@ void ensureClusterHasEncryptedMessage() { } ``` -The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest/feature` test. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. +The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest/feature` test. The upstream cluster is provisioned explicitly; the `KafkaClusterFixture` implementation (TestContainers, Strimzi, etc.) is selected by system property. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. ### What Operator Tests Look Like @@ -423,8 +453,9 @@ void shouldUpdateWhenFilterConfigurationChanges(String namespace) { ```java @Test void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { + KafkaClusterHandle upstream = kafkaClusterFixture.provision(); ProxyHandle proxy = operatorFixture.apply(ProxyScenario.builder() - .withUpstream(clusterName) + .withUpstream(upstream) .withFilter(new SimpleTransformFilterSpec("foo", "bar")) .build()); @@ -449,11 +480,11 @@ void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { ## Affected/Not Affected Projects **Affected:** -- **systemtest/feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `ProxyScenario`, `ProxyHandle`, `FilterSpec`). No Kubernetes dependency. +- **systemtest/feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `KafkaClusterFixture`, `ProxyScenario`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec`). No Kubernetes dependency. - **systemtest/operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. - **systemtest/webhook**: new module. Webhook behaviour tests migrated here. Depends on `KubernetesCapability`. - **systemtest/installer**: new module. One deployment smoke test, run across the installer/fixture matrix by CI. -- **systemtest (parent)**: parent module with child test modules (`feature`, `operator`, `webhook`, `installer`). Contains the shared framework (fixture interfaces, `ProxyScenario`, `ProxyHandle`, `KubernetesCapability`, fixture implementations) and a public API module for the `Installer` interface and supporting types. +- **systemtest (parent)**: parent module with child test modules (`feature`, `operator`, `webhook`, `installer`). Contains the shared framework (fixture interfaces, `ProxyScenario`, `ProxyHandle`, `KafkaClusterHandle`, `KubernetesCapability`, fixture implementations) and a public API module for the `Installer` and `KafkaClusterFixture` interfaces and supporting types. - **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest/operator`. **Not affected:** @@ -465,7 +496,7 @@ void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { This proposal introduces new framework abstractions alongside the existing code. Existing tests continue to work throughout the migration — the new layer wraps the existing `Kroxylicious` class internally. No test assertions change; only setup code is replaced. -The `ProxyFixture` and `Installer` interfaces are designed for extension. Downstream distributors typically implement `Installer` and compose it with upstream fixtures; distributors with fundamentally different deployment models implement `ProxyFixture` directly. Once published, `ProxyFixture`, `Installer`, `ProxyScenario`, and `ProxyHandle` become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. +The `ProxyFixture` and `Installer` interfaces are designed for extension. Downstream distributors typically implement `Installer` and compose it with upstream fixtures; distributors with fundamentally different deployment models implement `ProxyFixture` directly. Once published, `ProxyFixture`, `KafkaClusterFixture`, `Installer`, `ProxyScenario`, `ProxyHandle`, and `KafkaClusterHandle` become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. ## Rejected Alternatives From f025fe5537e2ac817b60fd1cc6533fc43f1e11c1 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 29 Jul 2026 13:20:43 +1200 Subject: [PATCH 15/15] refactor: rename ProxyScenario to ProxyDefinition, switch fixture selection to env vars, clarify reconfigure/waitForRestart semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename ProxyScenario -> ProxyDefinition throughout: the name implied a family of *Scenario siblings (OperatorScenario, WebhookScenario) that don't exist and shouldn't. ProxyDefinition reads as a one-off type and is a natural extension point for evolving from JSON blobs to a richer DSL. - Switch fixture/installer/client selection from system properties to environment variables (KROXYLICIOUS_TEST_FIXTURE, KROXYLICIOUS_TEST_INSTALLER, KROXYLICIOUS_TEST_KAFKA_CLUSTER, KROXYLICIOUS_TEST_CLIENT_DRIVER, KROXYLICIOUS_TEST_CLIENT_LOCATION). Env vars work uniformly across Maven (via Surefire), Gradle, and any CI tooling downstream distributors use. System properties are Java convention but add no practical value here. - Clarify that reconfigure() follows the same convergence contract as apply() — it blocks until the proxy has converged to the new config and returns a new ProxyHandle. The old handle should not be used after reconfigure(). - Expand waitForRestart() description: it is a high-level gesture for tests where the restart itself is the scenario under test (e.g. asserting clients reconnect seamlessly), not a convergence gate for config changes — that is reconfigure()'s job. Signed-off-by: Sam Barker --- proposals/111-system-test-framework.md | 68 ++++++++++++++------------ 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md index 2d411ff6..61b7f182 100644 --- a/proposals/111-system-test-framework.md +++ b/proposals/111-system-test-framework.md @@ -4,7 +4,7 @@ Introduce a layered abstraction for system tests that separates test intent from deployment mechanism, and organise tests into modules by what they cover: feature behaviour, operator reconciliation, webhook behaviour, and installation validation. -A `ProxyScenario` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. A `KafkaClusterFixture` provisions the upstream Kafka cluster and returns a `KafkaClusterHandle` — the same handle-based convergence pattern, making cluster provisioning pluggable across implementations (TestContainers, Strimzi, in-VM). An `Installer` — the primary downstream extension point — handles getting project components (operator, webhook, CRDs, RBAC) into the cluster independently of how proxies are deployed. +A `ProxyDefinition` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. A `KafkaClusterFixture` provisions the upstream Kafka cluster and returns a `KafkaClusterHandle` — the same handle-based convergence pattern, making cluster provisioning pluggable across implementations (TestContainers, Strimzi, in-VM). An `Installer` — the primary downstream extension point — handles getting project components (operator, webhook, CRDs, RBAC) into the cluster independently of how proxies are deployed. Feature tests become portable across deployment mechanisms — the same test runs against a CRD-deployed proxy, a manifest-managed proxy, a standalone process, or a downstream distribution — and cheap enough to write before the production code, as a specification. @@ -59,7 +59,7 @@ These are not tags on a single test suite — they are separate modules with dif | Module | Depends on | Kubernetes dependency | Portable across fixtures | |---|---|---|---| -| `systemtest/feature` | `ProxyFixture`, `KafkaClusterFixture`, `ProxyScenario`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec` | None | Yes — runs against any `ProxyFixture` | +| `systemtest/feature` | `ProxyFixture`, `KafkaClusterFixture`, `ProxyDefinition`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec` | None | Yes — runs against any `ProxyFixture` | | `systemtest/operator` | `OperatorFixture`, `KafkaClusterFixture`, `KubernetesCapability`, CRD types, K8s client | Yes | No — requires operator installed | | `systemtest/webhook` | `WebhookFixture`, `KubernetesCapability`, K8s client | Yes | No — requires webhook installed | | `systemtest/installer` | `ProxyFixture`, `KafkaClusterFixture`, `KubernetesCapability` | Yes (except standalone) | No — one test per installer | @@ -68,17 +68,17 @@ Feature tests do not import Kubernetes types. They cannot accidentally depend on All four modules are consumable as a TCK. A downstream distributor runs feature tests to prove their distribution satisfies the proxy's behavioural contract, operator tests to prove reconciliation works with their installation, webhook tests to prove admission webhook behaviour, and installer tests to prove their installation method works. -### `ProxyScenario` — Intent Without Deployment +### `ProxyDefinition` — Intent Without Deployment A plain Java value object describing what configuration the proxy should have. No knowledge of namespaces, CRD templates, or deployment mechanism. ```java -ProxyScenario scenario = ProxyScenario.builder() +ProxyDefinition scenario = ProxyDefinition.builder() .withUpstream(upstream) .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) .build(); -ProxyScenario scenario = ProxyScenario.builder() +ProxyDefinition scenario = ProxyDefinition.builder() .withUpstream(upstream) .withFilter(new RecordEncryptionFilterSpec(testKmsFacade) .withExperimentalConfig(config)) @@ -109,11 +109,11 @@ new RawFilterSpec("com.example.MyFilter", new MyFilterConfig(...)) ### `ProxyFixture` — Application and Convergence -The fixture translates a `ProxyScenario` into running infrastructure, blocks until the proxy has converged, and returns a `ProxyHandle`. +The fixture translates a `ProxyDefinition` into running infrastructure, blocks until the proxy has converged, and returns a `ProxyHandle`. ```java interface ProxyFixture { - ProxyHandle apply(ProxyScenario scenario); + ProxyHandle apply(ProxyDefinition scenario); } ``` @@ -138,13 +138,13 @@ We recognise there is likely to be a set of standard installers which need custo ### `ProxyFixture` Implementations -Tests never instantiate fixtures or installers — the JUnit extension reads system properties (`-Dfixture`, `-Dinstaller`) and composes them. The composition is extension-internal; the test sees only an injected fixture. +Tests never instantiate fixtures or installers — the JUnit extension reads environment variables (`KROXYLICIOUS_TEST_FIXTURE`, `KROXYLICIOUS_TEST_INSTALLER`) and composes them. The composition is extension-internal; the test sees only an injected fixture. Four `ProxyFixture` implementations cover the deployment mechanisms. Feature tests run against all of them unchanged — the test calls `apply(scenario)` and gets a `ProxyHandle` regardless of how the proxy was deployed. **`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. Calls `install(OPERATOR)` to put the operator into the cluster, then applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. -**`ManifestProxyFixture`**: translates `ProxyScenario` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. +**`ManifestProxyFixture`**: translates `ProxyDefinition` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. **`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. Calls `install(WEBHOOK)` to put the admission webhook into the cluster, then creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. @@ -174,14 +174,16 @@ The only way to obtain a `ProxyHandle` is through `ProxyFixture.apply()`. This m interface ProxyHandle { String bootstrap(); String bootstrap(ClientLocation location); - ProxyHandle reconfigure(ProxyScenario scenario); + ProxyHandle reconfigure(ProxyDefinition scenario); void waitForRestart(); } ``` `bootstrap()` defaults to `ClientLocation.ON_CLUSTER`. Tests using off-cluster clients call `bootstrap(ClientLocation.OFF_CLUSTER)` to obtain the externally accessible address; the fixture provides the right value for the deployment. -`waitForRestart()` is on `ProxyHandle` rather than on any capability because restarting the proxy is meaningful across all fixture types — on Kubernetes the fixture observes the Deployment rollout; on bare metal the fixture manages the process restart directly. The concept is universal; the mechanism is fixture-specific. +`reconfigure()` follows the same convergence contract as `apply()` — it blocks until the proxy has converged to the new configuration and returns a new `ProxyHandle`. The old handle should not be used after calling `reconfigure()`. + +`waitForRestart()` is on `ProxyHandle` rather than on any capability because restarting the proxy is meaningful across all fixture types — on Kubernetes the fixture observes the Deployment rollout; on bare metal the fixture manages the process restart directly. It is a high-level gesture: restart the proxy and block until it is back and serving. Tests that treat the restart itself as the scenario under test — e.g. asserting clients reconnect seamlessly — call `waitForRestart()` after triggering conditions that cause a restart. The concept is universal; the mechanism is fixture-specific. ### `KafkaClusterFixture` — Upstream Cluster Provisioning @@ -199,11 +201,11 @@ interface KafkaClusterHandle { } ``` -`KafkaClusterHandle` serves the same role as `ProxyHandle` — you cannot interact with the cluster before provisioning has completed. `ProxyScenario.withUpstream()` takes a `KafkaClusterHandle` rather than a raw string, making the dependency on a provisioned cluster explicit in the type system. +`KafkaClusterHandle` serves the same role as `ProxyHandle` — you cannot interact with the cluster before provisioning has completed. `ProxyDefinition.withUpstream()` takes a `KafkaClusterHandle` rather than a raw string, making the dependency on a provisioned cluster explicit in the type system. **Lifecycle is the test author's choice.** A `KafkaClusterHandle` provisioned in `@BeforeAll` is shared across all tests in the class — sensible when tests don't mutate cluster state and the cluster is expensive to provision. A handle provisioned in the test body is per-test. The extension manages cleanup at the appropriate scope. This is the same model as `ProxyFixture` — the framework provides the mechanism; the test author decides the scope. -**Selection**: the `KafkaClusterFixture` implementation is selected via system property (`-Dkafka.cluster=testcontainers|strimzi|invm`), consistent with fixture and installer selection. +**Selection**: the `KafkaClusterFixture` implementation is selected via the `KROXYLICIOUS_TEST_KAFKA_CLUSTER` environment variable, consistent with fixture and installer selection. Specifying the full set of `KafkaClusterFixture` implementations and their configuration surface is out of scope for this proposal. The key contribution is establishing that upstream cluster provisioning is a pluggable, fixture-managed concern with a handle-based convergence contract — not an ambient assumption. @@ -218,7 +220,7 @@ ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — **Module boundaries as compile-time separation**: `systemtest/feature` tests depend only on `ProxyFixture`, `KafkaClusterFixture`, and their handle types, and have no Kubernetes dependency. `systemtest/operator` tests depend on `OperatorFixture` and the Kubernetes client. `systemtest/webhook` tests depend on `WebhookFixture` and the Kubernetes client. These are compile-time boundaries that control what a test *can* depend on. -**Tags as runtime skip conditions**: module boundaries cannot account for runtime environment availability. Tags like `@Kubernetes` declare that a test requires a specific runtime environment and cause the extension to skip the test when that environment is not available — e.g. skipping `systemtest/operator` tests when no cluster is reachable. Fixture and installer selection is by system property (see [Fixture and Installer Selection](#fixture-and-installer-selection)). +**Tags as runtime skip conditions**: module boundaries cannot account for runtime environment availability. Tags like `@Kubernetes` declare that a test requires a specific runtime environment and cause the extension to skip the test when that environment is not available — e.g. skipping `systemtest/operator` tests when no cluster is reachable. Fixture and installer selection is by environment variable (see [Fixture and Installer Selection](#fixture-and-installer-selection)). **`KubernetesCapability`**: any test running on Kubernetes can have `KubernetesCapability` injected as a test parameter. This provides general-purpose access to the cluster environment — the namespace the proxy was deployed into and a `KubernetesClient` for observing resource state. Tests running on bare metal do not have `KubernetesCapability` available. @@ -230,6 +232,8 @@ ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — | `KafkaClusterHandle` | Always explicit via `provision()` | Provisioning is a blocking operation; must be visible | | `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | +**Symmetrical cleanup**: every install, provision, and apply action has a corresponding teardown action managed by the extension. The extension tracks the JUnit lifecycle scope at which each resource was created and performs cleanup at the symmetrical event — resources created in `@BeforeAll` or at class scope are torn down in `@AfterAll`; resources created in the test body are torn down after the test method. Test authors do not call explicit cleanup methods; the framework handles it. + ### `KubernetesCapability` — Cluster Environment Access `KubernetesCapability` provides access to the Kubernetes environment the proxy was deployed into. It is not operator-specific — it is available for any Kubernetes-backed fixture (operator, manifest, sidecar). @@ -241,31 +245,31 @@ interface KubernetesCapability { } ``` -The namespace is managed by the fixture. Each `apply()` call deploys into a namespace the fixture controls; the test discovers it through the capability rather than supplying it. This keeps `ProxyScenario` free of deployment concerns while giving tests the access they need for resource observation and client operations. +The namespace is managed by the fixture. Each `apply()` call deploys into a namespace the fixture controls; the test discovers it through the capability rather than supplying it. This keeps `ProxyDefinition` free of deployment concerns while giving tests the access they need for resource observation and client operations. ### Fixture and Installer Selection -Fixture, installer, and Kafka cluster provisioning are selected independently via system properties: +Fixture, installer, and Kafka cluster provisioning are selected independently via environment variables: ```bash # Default upstream: operator installed via manifests, proxy deployed via CRDs, TestContainers Kafka -mvn test -Dfixture=crd -Dinstaller=manifest -Dkafka.cluster=testcontainers +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=manifest KROXYLICIOUS_TEST_KAFKA_CLUSTER=testcontainers mvn test # OLM installation with Strimzi-managed Kafka -mvn test -Dfixture=crd -Dinstaller=olm -Dkafka.cluster=strimzi +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=olm KROXYLICIOUS_TEST_KAFKA_CLUSTER=strimzi mvn test # Downstream custom installer, upstream fixture -mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=com.example.downstream.MyInstaller mvn test # Manifest-managed proxy (no operator) -mvn test -Dfixture=manifest +KROXYLICIOUS_TEST_FIXTURE=manifest mvn test # Standalone with in-VM Kafka (fast local iteration) -mvn test -Dfixture=standalone -Dkafka.cluster=invm +KROXYLICIOUS_TEST_FIXTURE=standalone KROXYLICIOUS_TEST_KAFKA_CLUSTER=invm mvn test ``` -The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `-Dinstaller` is not specified, the fixture uses its default (manifest-based for fixtures that require an installer). Standalone and manifest fixtures do not take an installer. When `-Dkafka.cluster` is not specified, the extension uses a sensible default for the fixture type. +Environment variables work uniformly across Maven (via Surefire ``), Gradle, and any CI tooling a downstream distributor uses. The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `KROXYLICIOUS_TEST_INSTALLER` is not set, the fixture uses its default (manifest-based for fixtures that require an installer). Standalone and manifest fixtures do not take an installer. When `KROXYLICIOUS_TEST_KAFKA_CLUSTER` is not set, the extension uses a sensible default for the fixture type. ### `KafkaClient` Abstraction @@ -304,7 +308,7 @@ interface KafkaClient { The current produce/consume contract is sufficient for existing feature tests and is the starting point. The richer operations — transactions, consumer group management, admin — are the target shape. These are difficult or impossible to express via CLI-based drivers (kcat cannot commit a transaction mid-test), but fall out naturally from in-process drivers: the Java client directly, or librdkafka and Sarama via Java Foreign Function Interface. CLI drivers implement produce and consume; in-process drivers implement the full surface. -The current interface methods that are not test-intent (`inNamespace`, `getImage`, `preloadImage`) are execution-environment concerns and move out of the test-facing contract entirely. The framework wires the right driver and execution environment based on system properties (`-Dclient.driver=java|librdkafka|sarama`, `-Dclient.location=in-process|on-cluster`). +The current interface methods that are not test-intent (`inNamespace`, `getImage`, `preloadImage`) are execution-environment concerns and move out of the test-facing contract entirely. The framework wires the right driver and execution environment based on environment variables (`KROXYLICIOUS_TEST_CLIENT_DRIVER=java|librdkafka|sarama`, `KROXYLICIOUS_TEST_CLIENT_LOCATION=in-process|on-cluster`). This separation has a concrete payoff beyond cleanliness: Kroxylicious is a protocol proxy, and proving it works correctly with multiple client implementations — not just the Java client — is a first-class testing concern. A librdkafka or Sarama driver exercised via Java Foreign Function Interface from the test JVM would run in-process (fast, no pod startup cost) while proving protocol compatibility with a fundamentally different client implementation. The same feature test, unchanged, validates behaviour across client libraries. @@ -326,7 +330,7 @@ Where a fixture genuinely cannot run in a given environment — OLM absent, OCP ### Installer Tests -The `systemtest/installer` module contains a single smoke test: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. The test does not vary between installers — it is the same test run with different `-Dinstaller` and `-Dfixture` values. CI provides the matrix; the test provides the assertion. +The `systemtest/installer` module contains a single smoke test: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. The test does not vary between installers — it is the same test run with different `KROXYLICIOUS_TEST_INSTALLER` and `KROXYLICIOUS_TEST_FIXTURE` values. CI provides the matrix; the test provides the assertion. This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the `systemtest/feature` module; the installer test only asserts that the installation mechanism puts the proxy in a state where features can run. @@ -356,7 +360,7 @@ The framework provides public interfaces for downstream extensibility: `Installe Most downstream distributors differ only in how components are installed — their own OLM catalog, their own Helm chart, a different RBAC configuration. These distributors implement `Installer` (or configure an upstream-provided one) and compose it with upstream's fixtures, inheriting all deployment and convergence logic: ```bash -mvn test -Dfixture=crd -Dinstaller=com.example.downstream.MyInstaller +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=com.example.downstream.MyInstaller mvn test ``` Distributors with fundamentally different deployment models (e.g. a custom orchestrator, a managed service) may also need custom fixture implementations. The `Installer` and fixture interfaces have no upstream-specific dependencies in their signatures. @@ -411,7 +415,7 @@ void ensureClusterHasEncryptedMessage() { testKmsFacade.getTestKekManager().generateKek(KEK_PREFIX + topicName); KafkaClusterHandle upstream = kafkaClusterFixture.provision(); - ProxyHandle proxy = proxyFixture.apply(ProxyScenario.builder() + ProxyHandle proxy = proxyFixture.apply(ProxyDefinition.builder() .withUpstream(upstream) .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) .build()); @@ -424,7 +428,7 @@ void ensureClusterHasEncryptedMessage() { } ``` -The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest/feature` test. The upstream cluster is provisioned explicitly; the `KafkaClusterFixture` implementation (TestContainers, Strimzi, etc.) is selected by system property. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. +The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest/feature` test. The upstream cluster is provisioned explicitly; the `KafkaClusterFixture` implementation (TestContainers, Strimzi, etc.) is selected by environment variable. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. ### What Operator Tests Look Like @@ -454,7 +458,7 @@ void shouldUpdateWhenFilterConfigurationChanges(String namespace) { @Test void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { KafkaClusterHandle upstream = kafkaClusterFixture.provision(); - ProxyHandle proxy = operatorFixture.apply(ProxyScenario.builder() + ProxyHandle proxy = operatorFixture.apply(ProxyDefinition.builder() .withUpstream(upstream) .withFilter(new SimpleTransformFilterSpec("foo", "bar")) .build()); @@ -480,11 +484,11 @@ void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { ## Affected/Not Affected Projects **Affected:** -- **systemtest/feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `KafkaClusterFixture`, `ProxyScenario`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec`). No Kubernetes dependency. +- **systemtest/feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `KafkaClusterFixture`, `ProxyDefinition`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec`). No Kubernetes dependency. - **systemtest/operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. - **systemtest/webhook**: new module. Webhook behaviour tests migrated here. Depends on `KubernetesCapability`. - **systemtest/installer**: new module. One deployment smoke test, run across the installer/fixture matrix by CI. -- **systemtest (parent)**: parent module with child test modules (`feature`, `operator`, `webhook`, `installer`). Contains the shared framework (fixture interfaces, `ProxyScenario`, `ProxyHandle`, `KafkaClusterHandle`, `KubernetesCapability`, fixture implementations) and a public API module for the `Installer` and `KafkaClusterFixture` interfaces and supporting types. +- **systemtest (parent)**: parent module with child test modules (`feature`, `operator`, `webhook`, `installer`). Contains the shared framework (fixture interfaces, `ProxyDefinition`, `ProxyHandle`, `KafkaClusterHandle`, `KubernetesCapability`, fixture implementations) and a public API module for the `Installer` and `KafkaClusterFixture` interfaces and supporting types. - **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest/operator`. **Not affected:** @@ -496,7 +500,7 @@ void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { This proposal introduces new framework abstractions alongside the existing code. Existing tests continue to work throughout the migration — the new layer wraps the existing `Kroxylicious` class internally. No test assertions change; only setup code is replaced. -The `ProxyFixture` and `Installer` interfaces are designed for extension. Downstream distributors typically implement `Installer` and compose it with upstream fixtures; distributors with fundamentally different deployment models implement `ProxyFixture` directly. Once published, `ProxyFixture`, `KafkaClusterFixture`, `Installer`, `ProxyScenario`, `ProxyHandle`, and `KafkaClusterHandle` become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. +The `ProxyFixture` and `Installer` interfaces are designed for extension. Downstream distributors typically implement `Installer` and compose it with upstream fixtures; distributors with fundamentally different deployment models implement `ProxyFixture` directly. Once published, `ProxyFixture`, `KafkaClusterFixture`, `Installer`, `ProxyDefinition`, `ProxyHandle`, and `KafkaClusterHandle` become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. ## Rejected Alternatives @@ -515,7 +519,7 @@ We considered putting Kubernetes-specific methods (namespace, client access) dir ### Fixture selection via ServiceLoader -We considered using `ServiceLoader` to discover `ProxyFixture` implementations automatically from the classpath. This creates ambiguity when multiple fixture implementations are present (e.g. both OLM and Helm operator fixtures) and makes test runs non-deterministic. An explicit system property or Maven profile provides clear, reproducible fixture selection and composes naturally with CI matrix builds. +We considered using `ServiceLoader` to discover `ProxyFixture` implementations automatically from the classpath. This creates ambiguity when multiple fixture implementations are present (e.g. both OLM and Helm operator fixtures) and makes test runs non-deterministic. An explicit environment variable provides clear, reproducible fixture selection and composes naturally with CI matrix builds. ### Per-environment fixture implementations