diff --git a/proposals/111-system-test-framework.md b/proposals/111-system-test-framework.md new file mode 100644 index 0000000..61b7f18 --- /dev/null +++ b/proposals/111-system-test-framework.md @@ -0,0 +1,534 @@ +# 111 - System Test Framework + +## Summary + +Introduce a layered abstraction for system tests that separates test intent from deployment mechanism, and organise tests into modules by what they cover: feature behaviour, operator reconciliation, webhook behaviour, and installation validation. + +A `ProxyDefinition` describes the desired proxy configuration in deployment-agnostic terms; a `ProxyFixture` translates that into running infrastructure and blocks until convergence; the resulting `ProxyHandle` is a token of convergence that gates all subsequent interaction. A `KafkaClusterFixture` provisions the upstream Kafka cluster and returns a `KafkaClusterHandle` — the same handle-based convergence pattern, making cluster provisioning pluggable across implementations (TestContainers, Strimzi, in-VM). An `Installer` — the primary downstream extension point — handles getting project components (operator, webhook, CRDs, RBAC) into the cluster independently of how proxies are deployed. + +Feature tests become portable across deployment mechanisms — the same test runs against a CRD-deployed proxy, a manifest-managed proxy, a standalone process, or a downstream distribution — and cheap enough to write before the production code, as a specification. + +## Current Situation + +The system test suite covers the right things at the right level. Assertions in `RecordEncryptionST`, `AuthorizationST`, and `EntityIsolationST` test feature behaviour cleanly. `OperatorChangeDetectionST` tests operator reconciliation behaviour. The abstraction breaks down only in setup. + +Every feature test class has a private `deployXxx()` method that reimplements the same builder/template pattern against the operator's CRD types. Adding a new optional parameter (e.g. `ExperimentalKmsConfig`) requires touching every one of them. Timing workarounds are scattered across test classes with comments pointing at unresolved issues. The convergence question — "is the proxy actually serving the configuration I just applied?" — is answered by ad hoc polling in each test class rather than by a framework-level contract. + +This setup cost has a second-order effect: system tests are written after features merge, deferred until after merge and often left as an exercise to others because they are perceived as too expensive for a developer to include in a feature PR. The test framework is the bottleneck, not the assertions. + +The fix required is narrow: a thin setup layer that hides the deployment machinery without changing the assertions at all. + +## Motivation + +A system test is a precise, executable description of intended behaviour: deploy the proxy in configuration Y (Given), perturb it (When), assert the result (Then). If the framework makes that cheap to express, writing the system test first is feature-level TDD. The test fails until the production code makes it pass, and passing it is the definition of done. + +This model has already been validated in the codebase: the checksum annotation mechanism in the operator was built test-first. The system test described what the operator should do; the implementation followed. + +Three problems prevent this from being the norm: + +1. **Setup cost**: expressing "a proxy with this filter" requires navigating CRD builders, template classes, namespace management, and ingress configuration. The ceremony dwarfs the test. + +2. **No convergence contract**: the framework does not define when the proxy is ready. Each test class independently polls for readiness, with varying strategies and varying reliability. + +3. **Operator coupling**: every test implicitly requires the operator. Feature tests — which care only that a proxy has been started with a given configuration — cannot run without the full operator installation. This conflates feature correctness with operator correctness and prevents fast local iteration. + +Addressing these enables: + +- **Test-first development**: a developer writing a new filter can write a failing system test as the first commit of their feature branch, without reading framework documentation or asking for additional help. +- **Deployment-agnostic feature tests**: the same test runs against an operator-managed proxy, a manifest-managed proxy, or a Helm installation, with no changes to the test body. +- **Reliable convergence**: `proxyFixture.apply()` is a blocking call with a defined contract — when it returns, the proxy is serving the requested configuration. Manual polling disappears from test classes. +- **A Technology Compatibility Kit (TCK) for downstream distributions**: downstream distributors implement `Installer` for their distribution and run upstream's test modules — feature, operator, installer, and webhook — without forking. The test modules define what "correct" means; the distributor proves their packaging satisfies it. + +## Proposal + +### The Primary Seam + +The framework needs one organising question answered for every test class: **what is this test covering?** The answer determines which module the test belongs to — not which tag it carries, but which compile-time dependencies it has. + +Four categories of system test have fundamentally different concerns: + +- **Feature tests** — does record encryption work? Does authorisation enforce ACL rules? These tests care only that a proxy has been started with a given configuration. They must not care how the proxy was deployed. They have no Kubernetes dependency. + +- **Operator tests** — does the operator detect a configuration change and trigger a rolling restart? These tests are explicitly about the operator's reconciliation behaviour. They interact with Kubernetes resources directly and depend on the Kubernetes client. + +- **Webhook tests** — does the admission webhook inject sidecars into the right pods? Does it produce a valid pod spec? These tests are about the webhook's API interception behaviour, not proxy functionality. They depend on the Kubernetes client and the webhook being installed. + +- **Installer tests** — does a specific installation method (OLM, Helm, kustomize, standalone) produce a working proxy? These tests are about the installation mechanism, not the proxy features. The test does not vary between installers — CI provides the matrix. + +These are not tags on a single test suite — they are separate modules with different compile-time dependencies: + +| Module | Depends on | Kubernetes dependency | Portable across fixtures | +|---|---|---|---| +| `systemtest/feature` | `ProxyFixture`, `KafkaClusterFixture`, `ProxyDefinition`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec` | None | Yes — runs against any `ProxyFixture` | +| `systemtest/operator` | `OperatorFixture`, `KafkaClusterFixture`, `KubernetesCapability`, CRD types, K8s client | Yes | No — requires operator installed | +| `systemtest/webhook` | `WebhookFixture`, `KubernetesCapability`, K8s client | Yes | No — requires webhook installed | +| `systemtest/installer` | `ProxyFixture`, `KafkaClusterFixture`, `KubernetesCapability` | Yes (except standalone) | No — one test per installer | + +Feature tests do not import Kubernetes types. They cannot accidentally depend on CRDs, namespaces, or client libraries. The module boundary enforces this at compile time, not by convention. + +All four modules are consumable as a TCK. A downstream distributor runs feature tests to prove their distribution satisfies the proxy's behavioural contract, operator tests to prove reconciliation works with their installation, webhook tests to prove admission webhook behaviour, and installer tests to prove their installation method works. + +### `ProxyDefinition` — Intent Without Deployment + +A plain Java value object describing what configuration the proxy should have. No knowledge of namespaces, CRD templates, or deployment mechanism. + +```java +ProxyDefinition scenario = ProxyDefinition.builder() + .withUpstream(upstream) + .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) + .build(); + +ProxyDefinition scenario = ProxyDefinition.builder() + .withUpstream(upstream) + .withFilter(new RecordEncryptionFilterSpec(testKmsFacade) + .withExperimentalConfig(config)) + .withDownstreamTls(tls) + .build(); +``` + +Where `upstream` is a `KafkaClusterHandle` obtained from `kafkaClusterFixture.provision()` — see [KafkaClusterFixture](#kafkaclusterfixture--upstream-cluster-provisioning). + +### `FilterSpec` — The Filter DSL + +`FilterSpec` is an interface for expressing which filter the proxy should run and how it should be configured — in terms of the filter's purpose, not its deployment mechanics. No template classes, no filter type names, no namespaces visible to the test author. + +First-party filter types ship named implementations: + +```java +new RecordEncryptionFilterSpec(testKmsFacade) +new SimpleTransformFilterSpec("foo", "bar") +``` + +Custom filter authors implement `FilterSpec` directly for their own filter type, using whatever config model their filter defines. This is the same extension point a developer would use when writing a system test before the filter exists: the `FilterSpec` implementation is the specification. + +An escape hatch avoids the need for a dedicated class when one is not justified: + +```java +new RawFilterSpec("com.example.MyFilter", new MyFilterConfig(...)) +``` + +### `ProxyFixture` — Application and Convergence + +The fixture translates a `ProxyDefinition` into running infrastructure, blocks until the proxy has converged, and returns a `ProxyHandle`. + +```java +interface ProxyFixture { + ProxyHandle apply(ProxyDefinition scenario); +} +``` + +This is an explicit call — not magic JUnit injection — because the test author needs to understand that `apply()` is a blocking operation that includes convergence waiting. Hiding it behind injection would obscure the framework's most important contract. + +Test fixtures span two independent concerns: how infrastructure components are installed (operator, webhook, CRDs, RBAC) and what the test is exercising. The `Installer` handles the first concern; the fixture type handles the second. Feature tests use `ProxyFixture`, operator tests use `OperatorFixture`, and webhook tests use `WebhookFixture`. + +### `Installer` — Infrastructure Installation + +`Installer` handles getting project components — operator, webhook, CRDs, RBAC rules, ServiceAccounts — into the cluster. It is a **public interface** — the primary extension point for downstream distributors, who typically vary only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed. The `Installer` interface and supporting types live in a dedicated API module so that downstream distributors can depend on it without pulling in framework internals. + +```java +enum Component { OPERATOR, WEBHOOK } + +interface Installer { + void install(Component component); + void uninstall(Component component); +} +``` + +We recognise there is likely to be a set of standard installers which need customising only by simple parameters such as a Helm chart URI or an OLM bundle image. Specifying those implementations and their configuration surface is out of scope for this proposal. A downstream distributor could configure an upstream-provided implementation or implement the interface directly. + +### `ProxyFixture` Implementations + +Tests never instantiate fixtures or installers — the JUnit extension reads environment variables (`KROXYLICIOUS_TEST_FIXTURE`, `KROXYLICIOUS_TEST_INSTALLER`) and composes them. The composition is extension-internal; the test sees only an injected fixture. + +Four `ProxyFixture` implementations cover the deployment mechanisms. Feature tests run against all of them unchanged — the test calls `apply(scenario)` and gets a `ProxyHandle` regardless of how the proxy was deployed. + +**`CrdProxyFixture`**: takes an `Installer` as a constructor dependency. Calls `install(OPERATOR)` to put the operator into the cluster, then applies Kroxylicious CRDs (`KafkaProxy`, `VirtualKafkaCluster`, `KafkaProtocolFilter`) via Server-Side Apply, then waits for observable convergence signals — the controller has reconciled the resources and the Deployment has reached stable state with updated replicas ready and serving. The fixture knows the CRD schema and the convergence protocol, not the operator's internals. + +**`ManifestProxyFixture`**: translates `ProxyDefinition` into a proxy configuration file and a Kubernetes Deployment, applies them, then waits for the Deployment to reach stable state. No operator or installer required. + +**`SidecarProxyFixture`**: takes an `Installer` as a constructor dependency. Calls `install(WEBHOOK)` to put the admission webhook into the cluster, then creates a pod with the injection annotation, waits for the webhook to mutate it and the sidecar to be ready, then returns a `ProxyHandle`. Feature tests run against it unchanged — the proxy happens to be a sidecar rather than a standalone Deployment. + +**`StandaloneProxyFixture`**: starts the proxy as a local Java process with a generated configuration file, waits for the port to be ready, and returns a `ProxyHandle` with a localhost bootstrap. No Kubernetes, no installer, no namespaces. `KubernetesCapability` is not available for tests running against this fixture. + +Kubernetes fixtures use Server-Side Apply. Neither `CrdProxyFixture` nor `ManifestProxyFixture` requires `createOrUpdate` branching or `resourceVersion` management. + +**A note on convergence**: the framework waits for the best observable signal, not a guarantee. There is an inherent gap between "the operator updated the Deployment" and "the new pods are handling traffic." The `ProxyFixture` contract is: when `apply()` returns, the proxy is serving the requested configuration to the best observable precision. + +### `OperatorFixture` — Reconciliation Testing + +Operator tests are explicitly about the operator's behaviour — they need to manipulate CRDs and observe how the operator reconciles them. `OperatorFixture` calls `install(OPERATOR)` internally and exposes the operator's surface directly: applying CRD changes, observing reconciliation outcomes, and verifying that the operator reacts correctly to configuration changes, invalid inputs, and resource deletion. + +Tests in `systemtest/operator` receive an `OperatorFixture` rather than a `ProxyFixture`. The test drives the operator through its CRD API and asserts the resulting cluster state. + +### `WebhookFixture` — Admission Webhook Testing + +Webhook tests are about the admission webhook's mutation behaviour — they need to create pods and verify that the webhook mutates them correctly. `WebhookFixture` calls `install(WEBHOOK)` internally and exposes the webhook's surface: creating annotated pods, inspecting the resulting pod spec, and verifying that injection produces valid sidecar configurations. + +Tests in `systemtest/webhook` receive a `WebhookFixture` rather than a `ProxyFixture`. The test drives the webhook through pod creation and asserts the mutation outcome. + +### `ProxyHandle` — A Token of Convergence + +The only way to obtain a `ProxyHandle` is through `ProxyFixture.apply()`. This means a test cannot accidentally interact with the proxy before convergence has been waited for. + +```java +interface ProxyHandle { + String bootstrap(); + String bootstrap(ClientLocation location); + ProxyHandle reconfigure(ProxyDefinition scenario); + void waitForRestart(); +} +``` + +`bootstrap()` defaults to `ClientLocation.ON_CLUSTER`. Tests using off-cluster clients call `bootstrap(ClientLocation.OFF_CLUSTER)` to obtain the externally accessible address; the fixture provides the right value for the deployment. + +`reconfigure()` follows the same convergence contract as `apply()` — it blocks until the proxy has converged to the new configuration and returns a new `ProxyHandle`. The old handle should not be used after calling `reconfigure()`. + +`waitForRestart()` is on `ProxyHandle` rather than on any capability because restarting the proxy is meaningful across all fixture types — on Kubernetes the fixture observes the Deployment rollout; on bare metal the fixture manages the process restart directly. It is a high-level gesture: restart the proxy and block until it is back and serving. Tests that treat the restart itself as the scenario under test — e.g. asserting clients reconnect seamlessly — call `waitForRestart()` after triggering conditions that cause a restart. The concept is universal; the mechanism is fixture-specific. + +### `KafkaClusterFixture` — Upstream Cluster Provisioning + +Tests need an upstream Kafka cluster to proxy. Provisioning that cluster is a pluggable concern with meaningfully different implementations: an in-VM broker for fast local iteration, TestContainers for a real broker without a Kubernetes dependency, Strimzi for Kubernetes-native clusters, or a managed service. + +`KafkaClusterFixture` follows the same pattern as the other fixtures: the extension injects it, the test author calls `provision()` explicitly, and the returned `KafkaClusterHandle` is a token proving the cluster is ready. + +```java +interface KafkaClusterFixture { + KafkaClusterHandle provision(); +} + +interface KafkaClusterHandle { + String bootstrap(); +} +``` + +`KafkaClusterHandle` serves the same role as `ProxyHandle` — you cannot interact with the cluster before provisioning has completed. `ProxyDefinition.withUpstream()` takes a `KafkaClusterHandle` rather than a raw string, making the dependency on a provisioned cluster explicit in the type system. + +**Lifecycle is the test author's choice.** A `KafkaClusterHandle` provisioned in `@BeforeAll` is shared across all tests in the class — sensible when tests don't mutate cluster state and the cluster is expensive to provision. A handle provisioned in the test body is per-test. The extension manages cleanup at the appropriate scope. This is the same model as `ProxyFixture` — the framework provides the mechanism; the test author decides the scope. + +**Selection**: the `KafkaClusterFixture` implementation is selected via the `KROXYLICIOUS_TEST_KAFKA_CLUSTER` environment variable, consistent with fixture and installer selection. + +Specifying the full set of `KafkaClusterFixture` implementations and their configuration surface is out of scope for this proposal. The key contribution is establishing that upstream cluster provisioning is a pluggable, fixture-managed concern with a handle-based convergence contract — not an ambient assumption. + +### Injection Model and Tags + +The appropriate fixtures (`ProxyFixture`, `OperatorFixture`, `WebhookFixture`, and `KafkaClusterFixture`) are injected by the JUnit extension at class scope — they are environment configuration concerns, long-lived, with no timing implications. `ProxyHandle` and `KafkaClusterHandle` are always obtained explicitly by calling `apply()` or `provision()` in the test body. These calls are blocking and include convergence waiting; making them explicit ensures the test author understands the contract. + +```java +KafkaClusterHandle upstream = kafkaClusterFixture.provision(); // explicit — provisioning visible +ProxyHandle proxy = proxyFixture.apply(scenario); // explicit — convergence visible +``` + +**Module boundaries as compile-time separation**: `systemtest/feature` tests depend only on `ProxyFixture`, `KafkaClusterFixture`, and their handle types, and have no Kubernetes dependency. `systemtest/operator` tests depend on `OperatorFixture` and the Kubernetes client. `systemtest/webhook` tests depend on `WebhookFixture` and the Kubernetes client. These are compile-time boundaries that control what a test *can* depend on. + +**Tags as runtime skip conditions**: module boundaries cannot account for runtime environment availability. Tags like `@Kubernetes` declare that a test requires a specific runtime environment and cause the extension to skip the test when that environment is not available — e.g. skipping `systemtest/operator` tests when no cluster is reachable. Fixture and installer selection is by environment variable (see [Fixture and Installer Selection](#fixture-and-installer-selection)). + +**`KubernetesCapability`**: any test running on Kubernetes can have `KubernetesCapability` injected as a test parameter. This provides general-purpose access to the cluster environment — the namespace the proxy was deployed into and a `KubernetesClient` for observing resource state. Tests running on bare metal do not have `KubernetesCapability` available. + +| Concept | How obtained | Reason | +|---|---|---| +| `ProxyFixture` / `OperatorFixture` / `WebhookFixture` | Injected (class-scoped) | Environment config, no timing implications | +| `KafkaClusterFixture` | Injected (class-scoped) | Environment config, no timing implications | +| `KubernetesCapability` | Injected for Kubernetes-deployed tests | Namespace and client access for resource observation | +| `KafkaClusterHandle` | Always explicit via `provision()` | Provisioning is a blocking operation; must be visible | +| `ProxyHandle` | Always explicit via `apply()` | Convergence is a blocking operation; must be visible | + +**Symmetrical cleanup**: every install, provision, and apply action has a corresponding teardown action managed by the extension. The extension tracks the JUnit lifecycle scope at which each resource was created and performs cleanup at the symmetrical event — resources created in `@BeforeAll` or at class scope are torn down in `@AfterAll`; resources created in the test body are torn down after the test method. Test authors do not call explicit cleanup methods; the framework handles it. + +### `KubernetesCapability` — Cluster Environment Access + +`KubernetesCapability` provides access to the Kubernetes environment the proxy was deployed into. It is not operator-specific — it is available for any Kubernetes-backed fixture (operator, manifest, sidecar). + +```java +interface KubernetesCapability { + String namespace(); + KubernetesClient client(); +} +``` + +The namespace is managed by the fixture. Each `apply()` call deploys into a namespace the fixture controls; the test discovers it through the capability rather than supplying it. This keeps `ProxyDefinition` free of deployment concerns while giving tests the access they need for resource observation and client operations. + + +### Fixture and Installer Selection + +Fixture, installer, and Kafka cluster provisioning are selected independently via environment variables: + +```bash +# Default upstream: operator installed via manifests, proxy deployed via CRDs, TestContainers Kafka +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=manifest KROXYLICIOUS_TEST_KAFKA_CLUSTER=testcontainers mvn test + +# OLM installation with Strimzi-managed Kafka +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=olm KROXYLICIOUS_TEST_KAFKA_CLUSTER=strimzi mvn test + +# Downstream custom installer, upstream fixture +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=com.example.downstream.MyInstaller mvn test + +# Manifest-managed proxy (no operator) +KROXYLICIOUS_TEST_FIXTURE=manifest mvn test + +# Standalone with in-VM Kafka (fast local iteration) +KROXYLICIOUS_TEST_FIXTURE=standalone KROXYLICIOUS_TEST_KAFKA_CLUSTER=invm mvn test +``` + +Environment variables work uniformly across Maven (via Surefire ``), Gradle, and any CI tooling a downstream distributor uses. The extension composes them: it instantiates the installer, passes it to the fixture constructor, and manages the lifecycle. When `KROXYLICIOUS_TEST_INSTALLER` is not set, the fixture uses its default (manifest-based for fixtures that require an installer). Standalone and manifest fixtures do not take an installer. When `KROXYLICIOUS_TEST_KAFKA_CLUSTER` is not set, the extension uses a sensible default for the fixture type. + +### `KafkaClient` Abstraction + +The same separation of concerns that motivates the fixture model applies to how tests interact with Kafka. A test expresses intent — produce a message, consume from a topic, commit a transaction — without knowing which client library speaks the protocol or where that client runs. These are three independent axes: + +| Axis | What it answers | Examples | +|---|---|---| +| Test intent | What does the test want to do? | Produce, consume, create topic, transact | +| Client driver | Which protocol implementation? | Java client, librdkafka (via FFI), Sarama (via FFI), kcat CLI | +| Execution environment | Where does the client run? | In-process (test JVM), Kubernetes Job, local process | + +The test sees only intent. The framework composes driver and execution environment. CI can vary them independently — the same feature test runs against the Java client in-process, librdkafka via FFI, or kcat in a Kubernetes pod. + +The existing `KafkaClient` interface (StrimziTestClient, KcatClient, KafClient, PythonTestClient) conflates all three axes. Every implementation is a specific driver running as a Kubernetes Job. The interface itself mixes the produce/consume contract with Kubernetes-specific machinery: + +```java +KafkaClient inNamespace(String namespace); +String getImage(); +void preloadImage(); +``` + +The proposed model separates them. `KafkaClient` becomes a richer test-facing API — pure intent, expressed in terms of Kafka operations rather than CLI invocations: + +```java +interface KafkaClient { + void produce(String topic, String bootstrap, List records); + List consume(String topic, String bootstrap, int count); + void createTopic(String topic, String bootstrap, int partitions, int replicas); + + // Richer operations — enabled by in-process drivers + void produceInTransaction(String topic, String bootstrap, List records); + void commitOffsets(String groupId, String bootstrap, Map offsets); + AdminClient admin(String bootstrap); +} +``` + +The current produce/consume contract is sufficient for existing feature tests and is the starting point. The richer operations — transactions, consumer group management, admin — are the target shape. These are difficult or impossible to express via CLI-based drivers (kcat cannot commit a transaction mid-test), but fall out naturally from in-process drivers: the Java client directly, or librdkafka and Sarama via Java Foreign Function Interface. CLI drivers implement produce and consume; in-process drivers implement the full surface. + +The current interface methods that are not test-intent (`inNamespace`, `getImage`, `preloadImage`) are execution-environment concerns and move out of the test-facing contract entirely. The framework wires the right driver and execution environment based on environment variables (`KROXYLICIOUS_TEST_CLIENT_DRIVER=java|librdkafka|sarama`, `KROXYLICIOUS_TEST_CLIENT_LOCATION=in-process|on-cluster`). + +This separation has a concrete payoff beyond cleanliness: Kroxylicious is a protocol proxy, and proving it works correctly with multiple client implementations — not just the Java client — is a first-class testing concern. A librdkafka or Sarama driver exercised via Java Foreign Function Interface from the test JVM would run in-process (fast, no pod startup cost) while proving protocol compatibility with a fundamentally different client implementation. The same feature test, unchanged, validates behaviour across client libraries. + +**Bootstrap address pairing**: an off-cluster client needs an externally accessible bootstrap address, not the cluster-internal Service DNS. This pairs naturally with `ProxyHandle.bootstrap(ClientLocation)` — the framework wires client location to bootstrap address at test setup time. The test author calls `proxy.bootstrap()` and receives the right address for the client that is configured. + +### Cluster Environment + +The framework must be runnable across three meaningfully different environments: + +| Environment | Characteristics | +|---|---| +| Minikube / local K8s | Local dev path. No OLM. Limited networking. Fast iteration. | +| Vanilla remote K8s | CI path. OLM optional. LoadBalancer or NodePort ingress. | +| OpenShift (OCP) | OLM native. Routes instead of Ingress. Security Context Constraints. | + +The principle is that **environment differences are absorbed by the fixture, not exposed to the test**. A `CrdProxyFixture` on OCP creates a Route; on vanilla K8s it creates a LoadBalancer Service. The test sees only `proxy.bootstrap()`. Cluster environment is a constructor-time or environment-variable-time concern for the fixture implementation. + +Where a fixture genuinely cannot run in a given environment — OLM absent, OCP required — it throws `AssumptionViolatedException` and the test skips. This is the same skip mechanism, extended to cluster environment. A test run on minikube naturally skips OLM deployment tests and any OCP-specific webhook behaviour tests without configuration. + +### Installer Tests + +The `systemtest/installer` module contains a single smoke test: deploy a proxy with a file-based filter (one that reads substitution values from a mounted file), produce a message, assert the consumer sees the transformed value. The test does not vary between installers — it is the same test run with different `KROXYLICIOUS_TEST_INSTALLER` and `KROXYLICIOUS_TEST_FIXTURE` values. CI provides the matrix; the test provides the assertion. + +This test is deliberately minimal — it is not a feature matrix. Its purpose is to catch installation failures: the plugin does not load, the Secret is not mounted, the file path is wrong. Features are correct by virtue of the `systemtest/feature` module; the installer test only asserts that the installation mechanism puts the proxy in a state where features can run. + +Each installer test run has a single reason to fail: the consumer did not see the transformed value. Every possible installation failure collapses into that one observable. No separate assertions per failure mode are needed or wanted; they all manifest identically, and the CI matrix entry tells you which installation mechanism failed. + +| Install method | Fixture | Installer | File config mechanism | +|---|---|---|---| +| CRD (manifests) | `CrdProxyFixture` | manifest-based | Kubernetes Secret mount | +| CRD (Helm) | `CrdProxyFixture` | Helm-based | Kubernetes Secret mount | +| CRD (OLM) | `CrdProxyFixture` | OLM-based | Kubernetes Secret mount | +| Manifest (no operator) | `ManifestProxyFixture` | — | Kubernetes Secret mount | +| Sidecar injection (webhook) | `SidecarProxyFixture` | manifest-based | Kubernetes Secret mount | +| Standalone | `StandaloneProxyFixture` | — | File written to local path | + +### Webhook Tests + +The webhook touches two modules: + +**As a deployment path** (`systemtest/installer`): `SidecarProxyFixture` is an installer test entry in the matrix — it proves that sidecar injection produces a working proxy. The same single smoke test runs against it. Feature tests in `systemtest/feature` also run against `SidecarProxyFixture` unchanged. + +**As behaviour under test** (`systemtest/webhook`): a separate module that uses `WebhookFixture` to assert on the webhook's API interception behaviour. These tests ask questions that do not produce a `ProxyHandle`: does the webhook inject into pods with annotation X but not Y? Does it produce a valid pod spec? What happens when the webhook is unavailable and `failurePolicy: Ignore`? + +### TCK Extension Points + +The framework provides public interfaces for downstream extensibility: `Installer`, `ProxyFixture`, `OperatorFixture`, `WebhookFixture`, and `KafkaClusterFixture`. + +Most downstream distributors differ only in how components are installed — their own OLM catalog, their own Helm chart, a different RBAC configuration. These distributors implement `Installer` (or configure an upstream-provided one) and compose it with upstream's fixtures, inheriting all deployment and convergence logic: + +```bash +KROXYLICIOUS_TEST_FIXTURE=crd KROXYLICIOUS_TEST_INSTALLER=com.example.downstream.MyInstaller mvn test +``` + +Distributors with fundamentally different deployment models (e.g. a custom orchestrator, a managed service) may also need custom fixture implementations. The `Installer` and fixture interfaces have no upstream-specific dependencies in their signatures. + +All four test modules are consumable as a TCK: + +- **`systemtest/feature`**: downstream proves their distribution satisfies the proxy's behavioural contract — features work regardless of installation method. +- **`systemtest/installer`**: downstream proves their installation method produces a working system — their OLM catalog installs correctly, their Helm chart renders valid resources. +- **`systemtest/operator`**: downstream proves operator reconciliation works with their installation — change detection, status conditions, rolling restarts all function correctly. +- **`systemtest/webhook`**: downstream proves admission webhook behaviour works with their installation — sidecar injection targets the right pods, produces valid pod specs, and respects failure policies. + +Upstream maintains the definition of correct behaviour across all four modules. Most downstream distributors need only implement `Installer` for their installation mechanism. Distributors with fundamentally different deployment models may also need custom fixture implementations. + +### What Feature Tests Look Like + +The before/after comparison illustrates the effect of the abstraction on a representative test. + +**Before** (current `RecordEncryptionST`): + +```java +private void deployPortIdentifiesNodeWithRecordEncryptionFilter( + TestKmsFacade testKmsFacade, ExperimentalKmsConfig config) { + String filterName = KROXYLICIOUS_ENCRYPTION_FILTER_NAME + "-" + + testKmsFacade.getKmsServiceClass().getSimpleName().toLowerCase(); + kroxylicious = new KroxyliciousBuilder() + .withNamespace(Constants.KROXYLICIOUS_NAMESPACE) + .withKafkaProxy(KroxyliciousKafkaProxyTemplates + .defaultKafkaProxyCR(KROXYLICIOUS_PROXY_SIMPLE_NAME, 1).build()) + .withKafkaProxyIngress(KroxyliciousKafkaProxyIngressTemplates + .defaultKafkaProxyIngressCR(KROXYLICIOUS_INGRESS_CLUSTER_IP, + KROXYLICIOUS_PROXY_SIMPLE_NAME).build()) + .withKafkaService(KroxyliciousKafkaClusterRefTemplates + .defaultKafkaClusterRefCR(clusterName).build()) + .addKafkaProtocolFilter(KroxyliciousFilterTemplates + .kroxyliciousRecordEncryptionFilter(KROXYLICIOUS_NAMESPACE, + filterName, testKmsFacade, config).build()) + .withVirtualKafkaCluster(KroxyliciousVirtualKafkaClusterTemplates + .virtualKafkaClusterWithFilterCR(clusterName, + KROXYLICIOUS_PROXY_SIMPLE_NAME, clusterName, + KROXYLICIOUS_INGRESS_CLUSTER_IP, + List.of(filterName)).build()) + .build(); + kroxylicious.createOrUpdateResources(); +} +``` + +**After**: + +```java +@Test +void ensureClusterHasEncryptedMessage() { + testKmsFacade.getTestKekManager().generateKek(KEK_PREFIX + topicName); + + KafkaClusterHandle upstream = kafkaClusterFixture.provision(); + ProxyHandle proxy = proxyFixture.apply(ProxyDefinition.builder() + .withUpstream(upstream) + .withFilter(new RecordEncryptionFilterSpec(testKmsFacade)) + .build()); + + kafkaClient.createTopic(topicName, proxy.bootstrap(), 1, 1); + kafkaClient.produceMessages(topicName, proxy.bootstrap(), MESSAGE, 1); + + var consumed = kafkaClient.consumeMessages(topicName, proxy.bootstrap(), 1); + assertThat(consumed).allMatch(r -> !r.getPayload().contains(MESSAGE)); +} +``` + +The test contains only the Given/When/Then relevant to record encryption. No Kubernetes imports, no namespace — this is a `systemtest/feature` test. The upstream cluster is provisioned explicitly; the `KafkaClusterFixture` implementation (TestContainers, Strimzi, etc.) is selected by environment variable. It works against a CRD-deployed proxy, a manifest-managed proxy, a sidecar, or a standalone process. It can be written before the filter exists — it will fail (correctly) until the production code makes it pass. + +### What Operator Tests Look Like + +**Before** (current `OperatorChangeDetectionST`): + +```java +@Test +void shouldUpdateWhenFilterConfigurationChanges(String namespace) { + resourceManager.createOrUpdateResourceFromBuilderWithWait(arbitraryFilterBuilder); + deployPortIdentifiesNodeWithFilters(namespace, kafkaClusterName, List.of("arbitrary-filter")); + + var originalChecksum = getInitialChecksum(namespace); // ~30 lines of polling + var replacementConfig = Map.of("transformation", "Replacing", + "transformationConfig", Map.of("findPattern", "foo", "replacementValue", "updated")); + + resourceManager.replaceResourceWithRetries(arbitraryFilter, current -> { + current.getSpec().setConfigTemplate(replacementConfig); + }); + + assertDeploymentUpdated(namespace, originalChecksum); // polls checksum annotation +} +``` + +**After**: + +```java +@Test +void shouldUpdateWhenFilterConfigurationChanges(KubernetesCapability kube) { + KafkaClusterHandle upstream = kafkaClusterFixture.provision(); + ProxyHandle proxy = operatorFixture.apply(ProxyDefinition.builder() + .withUpstream(upstream) + .withFilter(new SimpleTransformFilterSpec("foo", "bar")) + .build()); + + String beforeChecksum = readChecksumAnnotation(kube.client(), kube.namespace()); + + kube.client().resources(KafkaProtocolFilter.class) + .inNamespace(kube.namespace()) + .withName(filterName) + .edit(current -> { + current.getSpec().setConfigTemplate(replacementConfig); + return current; + }); + + await().until( + () -> readChecksumAnnotation(kube.client(), kube.namespace()), + not(equalTo(beforeChecksum))); +} +``` + +`getInitialChecksum` disappears: `proxyFixture.apply()` blocks until convergence, so `readChecksumAnnotation()` is called against stable state — no polling required to establish the baseline. This test lives in the `systemtest/operator` module, which uses `OperatorFixture` and has compile-time access to `KubernetesCapability` and the CRD types. The resource mutation is intentionally direct — this test exists to prove the operator detects and responds to changes made outside the fixture. + +## Affected/Not Affected Projects + +**Affected:** +- **systemtest/feature**: new module. Feature tests migrated here. Depends only on the framework abstractions (`ProxyFixture`, `KafkaClusterFixture`, `ProxyDefinition`, `ProxyHandle`, `KafkaClusterHandle`, `FilterSpec`). No Kubernetes dependency. +- **systemtest/operator**: new module. Operator behaviour tests (`OperatorChangeDetectionST`) migrated here. Depends on `KubernetesCapability` and CRD types. +- **systemtest/webhook**: new module. Webhook behaviour tests migrated here. Depends on `KubernetesCapability`. +- **systemtest/installer**: new module. One deployment smoke test, run across the installer/fixture matrix by CI. +- **systemtest (parent)**: parent module with child test modules (`feature`, `operator`, `webhook`, `installer`). Contains the shared framework (fixture interfaces, `ProxyDefinition`, `ProxyHandle`, `KafkaClusterHandle`, `KubernetesCapability`, fixture implementations) and a public API module for the `Installer` and `KafkaClusterFixture` interfaces and supporting types. +- **kroxylicious-operator**: no code changes, but operator-managed system tests move to `systemtest/operator`. + +**Not affected:** +- **kroxylicious-proxy (runtime)**: no production code changes. The framework abstracts over the proxy; it does not change it. +- **kroxylicious-api**: the filter SPI is unaffected. +- **kroxylicious-kms and plugin modules**: no changes needed. + +## Compatibility + +This proposal introduces new framework abstractions alongside the existing code. Existing tests continue to work throughout the migration — the new layer wraps the existing `Kroxylicious` class internally. No test assertions change; only setup code is replaced. + +The `ProxyFixture` and `Installer` interfaces are designed for extension. Downstream distributors typically implement `Installer` and compose it with upstream fixtures; distributors with fundamentally different deployment models implement `ProxyFixture` directly. Once published, `ProxyFixture`, `KafkaClusterFixture`, `Installer`, `ProxyDefinition`, `ProxyHandle`, and `KafkaClusterHandle` become API surface for downstream consumers — their signatures should be treated as a compatibility commitment. + +## Rejected Alternatives + +### Magic JUnit injection of `ProxyHandle` + +We considered having the JUnit extension inject `ProxyHandle` directly as a test parameter (similar to how `ProxyFixture` is injected), with convergence waiting happening transparently during parameter resolution. This hides the most important contract in the framework — that `apply()` is a blocking operation — behind invisible lifecycle callbacks. The test author would not see when convergence happens, making it harder to reason about test timing and harder to debug when convergence fails. The explicit `proxyFixture.apply()` call keeps the blocking operation visible. + +### Unified `ProxyFixture.apply()` for both feature and operator tests + +We considered having operator tests use `proxyFixture.apply()` for all mutations, including the mid-test configuration changes that operator tests need to assert on. However, operator tests exist specifically to prove that the operator detects mutations made outside the fixture — bypassing the fixture for the mid-test mutation is the point of the test. Routing those mutations through the fixture would test the fixture's update path, not the operator's reconciliation behaviour. + + +### Merging `KubernetesCapability` into `ProxyHandle` + +We considered putting Kubernetes-specific methods (namespace, client access) directly on `ProxyHandle`. This would make `ProxyHandle` Kubernetes-aware, breaking its deployment-agnostic contract — a bare metal `ProxyHandle` has no namespace. Keeping them separate means every method on `ProxyHandle` is meaningful for every fixture type. + +### Fixture selection via ServiceLoader + +We considered using `ServiceLoader` to discover `ProxyFixture` implementations automatically from the classpath. This creates ambiguity when multiple fixture implementations are present (e.g. both OLM and Helm operator fixtures) and makes test runs non-deterministic. An explicit environment variable provides clear, reproducible fixture selection and composes naturally with CI matrix builds. + +### Per-environment fixture implementations + +We considered separate fixture classes for each cluster environment (e.g. `MinikubeCrdProxyFixture`, `OCPCrdProxyFixture`). This creates a combinatorial explosion of fixture classes and pushes environment-specific logic into class hierarchies. Environment differences are narrower than deployment-mechanism differences: the same `CrdProxyFixture` can create a Route on OCP and a LoadBalancer Service on vanilla K8s based on a constructor-time environment flag. The fixture class models the deployment mechanism; environment variation is configuration within that class. + +### `Installer` as extension-internal + +We considered hiding `Installer` as an extension-internal interface, with downstream distributors implementing the full `ProxyFixture`. However, downstream typically varies only by installation method (their own OLM catalog, their own Helm chart) and not by how proxies are deployed or converged. Making `Installer` public lets downstream write a single class and compose it with upstream's fixture, inheriting all proxy deployment and convergence logic. Forcing them to reimplement `ProxyFixture` for a difference that is entirely in the installation dimension wastes effort and risks divergence. + +### Abstract base class instead of `ProxyFixture` interface + +We considered providing an abstract base class with shared convergence-waiting logic rather than a plain interface. This would simplify fixture implementations at the cost of coupling them to the upstream base class. A downstream distributor who needs different convergence behaviour (e.g. polling a proprietary health endpoint) would need to override carefully or bypass the base class entirely. The plain interface is a cleaner extension point — shared convergence utilities can be offered as composition rather than inheritance. \ No newline at end of file