From 6c7e4b0eb2b2e474b088670feecccb5884088da2 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Sun, 19 Jul 2026 23:07:34 +0000 Subject: [PATCH 01/52] docs(proposal): add SASL termination design proposal Covers motivation, filter design with sealed state machine, mechanism handler extension point, OAUTHBEARER and SCRAM implementations, credential store SPI, security model, and rejected alternatives. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 339 ++++++++++++++++++++++++++++++ 1 file changed, 339 insertions(+) create mode 100644 proposals/000-sasl-termination.md diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md new file mode 100644 index 00000000..8443ab6b --- /dev/null +++ b/proposals/000-sasl-termination.md @@ -0,0 +1,339 @@ +# 000 - SASL Termination + +SASL termination allows the Kroxylicious proxy to authenticate Kafka clients directly, without forwarding SASL exchanges to the upstream Kafka broker. This enables credential isolation, authentication protocol translation, and centralized credential management. + +## Current situation + +Kroxylicious currently handles client SASL authentication in a number of ways: + +1. **SASL Passthrough**: The proxy forwards SASL exchanges unmodified between client and broker. The broker performs all authentication. + +2. **SASL Passthrough Inspection**: The [SASL inspection filter][sasl-inspection] observes SASL exchanges as they pass through, extracting the client's authorization ID without making authentication decisions itself. This supports SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and PLAIN mechanisms. + +3. **OAUTHBEARER Validation**: The [OAUTHBEARER validation filter][oauthbearer-validation] validates JWT tokens before forwarding `SaslAuthenticate` requests to the broker. This is a partial termination — it rejects invalid tokens early but still requires the broker to perform the actual SASL exchange for valid tokens. + +None of these approaches allow the proxy to fully terminate SASL: authenticating clients against its own credential store without any SASL interaction with the broker. + +[Proposal 004][proposal-004] defined the term "SASL Termination" as: _"a component that responds to a client's `SaslAuthenticate` requests itself, without forwarding those requests to the server."_ + +[Proposal 006][proposal-006] added the `clientSaslAuthenticationSuccess()` and `clientSaslAuthenticationFailure()` methods to the `FilterContext` API, and explicitly listed "Implement a 1st party SaslTerminator filter" as future work. + +This proposal realizes that future work. + +## Motivation + +### Credential isolation + +With SASL termination, the proxy authenticates clients using credentials stored in its own credential store. The broker never sees client credentials. This is valuable when: + +- Client credentials should not be shared with or managed by the Kafka cluster administrators. +- Different credential lifecycles are needed for client-facing and broker-facing authentication. +- Compliance requirements mandate credential isolation between organizational boundaries. + +### Authentication protocol translation + +The proxy can authenticate clients using one SASL mechanism (e.g. SCRAM-SHA-256) while using an entirely different authentication mechanism to connect to the broker (e.g. mTLS, or a service account). This enables: + +- Migrating broker authentication without changing client configurations. +- Using client-friendly mechanisms even when the broker supports only a limited set. +- Integrating with identity providers that don't have native Kafka client support. + +### Zero-trust edge authentication + +In a zero-trust architecture the proxy can enforce authentication at the network edge before any Kafka protocol traffic reaches brokers. Unauthenticated clients are rejected immediately, reducing the broker's attack surface. + +### Centralized credential management + +A single credential store serves all proxy instances, rather than requiring per-broker credential configuration. Combined with the proxy's existing plugin system, this allows integration with enterprise credential stores. + +### Broker-less authentication + +A key problem with any passthrough-based technique is that it depends on the availability of a specific Kafka cluster. With the advent of the routing API described by [Proposal 072][proposal-072] there is a need to be able to authenticate a client session before a connection has been made to any target cluster. This is unavoidable because the identity of the client might be an input to the subsequent routing decisions. + +## Proposal + +This proposal aims to support for the following SASL mechanisms: `SCRAM-SHA-256`, `SCRAM-SHA-512` and `OAUTHBEARER`. +It also aims to be flexible, so as to allow other mechanisms to be supported either in the future, or as plugins. + +### The filter + +The SASL termination filter intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, authenticating clients at the proxy and short-circuiting the responses without forwarding them to the broker. It enforces a security barrier: until a client has successfully authenticated, the only requests permitted are `API_VERSIONS`, `SASL_HANDSHAKE`, and `SASL_AUTHENTICATE`. All other request types are rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. + +#### State machine + +The filter maintains per-connection state using a sealed interface `State` with four concrete states: + +``` +RequiringHandshake ──→ RequiringAuthenticate ←──╮ + │ │ + ├─ (multi-round) ──╯ + │ + ├──→ Authenticated (terminal) + │ + └──→ Failed (terminal) +``` + +- **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests, which negotiate the mechanism and transition to `RequiringAuthenticate`. +- **RequiringAuthenticate:** Accepts `SASL_AUTHENTICATE` requests. Loops back to itself for multi-round mechanisms (e.g. SCRAM). Carries a reference to the `MechanismHandler` for the negotiated mechanism. +- **Authenticated:** Terminal success state. The filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters, then forwards all subsequent requests. +- **Failed:** Terminal failure state. The connection is closed. + +The sealed interface prevents creation of invalid states at compile time. + +### Mechanism handler extension point + +The filter delegates the actual authentication exchange to mechanism-specific handlers, discovered via an internal extension point: + +- `MechanismHandler` — handles the authentication exchange for a single connection. Implementations process `SaslAuthenticate` request bytes and return `AuthenticationResult` (CHALLENGE, SUCCESS, or FAILURE). Handlers are per-connection and not thread-safe. + +- `MechanismHandlerFactory` — manages mechanism-specific resources and creates handler instances. Discovered via `ServiceLoader`. Each factory: + 1. Reports its IANA-registered mechanism name. + 2. Receives mechanism-specific configuration at `initialize()` time and creates whatever resources the mechanism requires (credential stores, JWKS callback handlers, etc.). + 3. Creates per-connection `MechanismHandler` instances, injecting shared resources. + 4. Releases resources on `close()`. + +These are **not** user-facing plugins (no `@Plugin` annotation). They provide internal extensibility for adding new mechanism support without modifying the filter itself. +The intention behind the decision **not** to make these user-facing plugins is to encourage a small number of secure, high-quality implementations, one for each mechanism. +Allowing pluggable implementations would make auditing for correctness and security significantly harder. + +**Initial mechanism support:** + +| Mechanism | Handler | Notes | +|-----------|---------|-------| +| SCRAM-SHA-256 | `ScramHandler` via `ScramSha256HandlerFactory` | RFC 5802 | +| SCRAM-SHA-512 | `ScramHandler` via `ScramSha512HandlerFactory` | RFC 5802 | +| OAUTHBEARER | `OauthBearerHandler` via `OauthBearerHandlerFactory` | RFC 6750 / RFC 7628 | + +### OAUTHBEARER implementation + +The OAUTHBEARER handler validates JWT bearer tokens at the proxy without forwarding them to the broker. + +The handler uses Kafka's `OAuthBearerValidatorCallbackHandler` for JWT validation, the same mechanism used by the existing OAUTHBEARER validation filter. The `OauthBearerHandlerFactory` manages the JWKS endpoint configuration and callback handler lifecycle: at `initialize()`-time it configures the callback handler with the JWKS endpoint, expected audience/issuer, and refresh settings; per-connection handlers receive the shared callback handler and use it to create a `SaslServer` via the JSSE/SASL framework. + +OAUTHBEARER is architecturally the simpler mechanism — it requires no credential store. The factory's only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). + +**Key differences from the existing OAUTHBEARER validation filter:** +- The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, the termination handler validates tokens and _short-circuits_ — the broker never sees a SASL exchange. +- The handler factory owns its callback handler and JWKS configuration, receiving them at `initialize()`-time rather than requiring a credential store. + +### SCRAM implementation + +SCRAM is more complex than OAUTHBEARER because it is a multi-round challenge-response protocol that requires stored credentials. + +The SCRAM handler delegates to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework: + +1. **First round:** Extract the username from the SCRAM client-first-message, asynchronously look up the credential from the store, create a `SaslServer` with a `CallbackHandler` that supplies the credential, and process the first message. + +2. **Subsequent rounds:** Process messages through the existing `SaslServer` synchronously. When `SaslServer.isComplete()` returns true, return SUCCESS with the authorization ID from `SaslServer.getAuthorizationID()`. + +This approach avoids reimplementing the SCRAM protocol and benefits from Kafka's battle-tested implementation. + +#### SCRAM Credential store SPI + +SCRAM mechanisms need a way to look up stored credentials. The credential store SPI provides async credential lookup, decoupled from any particular storage backend. + +**Core types:** + +- `ScramCredentialStore` — the lookup interface, returning `CompletionStage` for a given username. Returns `null` (via completed stage) when the user is not found. Exceptional completions indicate infrastructure failures. + +- `ScramCredentialStoreService` — the lifecycle interface for credential store providers. Follows the initialize/build/close pattern used by `KmsService`: + 1. `initialize(C config)` — validate and store configuration. + 2. `buildCredentialStore()` — create an operational store instance. + 3. `close()` — release resources. + +- `ScramCredential` — an immutable sealed record holding the username, salt, iteration count, server key, stored key, and hash algorithm. Byte array fields use defensive copies in the constructor and accessors to prevent mutation. The `toString()` method redacts sensitive fields. + +- Exception hierarchy: `CredentialLookupException` with subtypes `CredentialServiceUnavailableException` and `CredentialServiceTimeoutException`. + +**Design note:** The SPI is intentionally SCRAM-specific. OAUTHBEARER uses token validation against a JWKS endpoint, which has a fundamentally different shape from stored credential lookup. Rather than creating a leaky abstraction that covers both, each mechanism family uses its own resource management approach (see [Rejected alternatives](#rejected-alternatives)). + +#### `KeyStore`-based credential store provider + +The first-party provider stores SCRAM credentials in a Java `KeyStore` file, following the project's established pattern of using `KeyStores` to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry keyed by username. + +**Characteristics:** + +- Loads the entire KeyStore into memory at construction time for sub-millisecond lookups. +- Does not support hot reloading — credential changes require a proxy restart or virtual cluster reconfiguration. +- Supports PKCS12 and JKS store types. +- Uses the Kroxylicious `PasswordProvider` abstraction for KeyStore and key passwords, supporting both file-based (production) and inline (development) password configuration. + +**CLI credential management tool** (`KeystoreCredentialTool`): + +The credentials stored in the KeyStore are serialized JSON, which makes for less than ideal UX: The user needs ensure the JSON has the required format. +Moreover, the values of that JSON are not all obvious things like the username. Some of the fields are computed from cryptographic operation on the password which need to +be done correctly for the authentication to work, and where incorrect construction can undermine security. + +To provide a better UX and to reduce the possibility of user error compromising security a PicoCLI-based command-line tool will be provided for managing credentials in KeyStore files. Supports: `create`, `add-user`, `remove-user`, `update-password`, `list-users`. + +Security measures: +- Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure. Command-line password arguments are supported, but gated behind an `--unlock-insecure-options` flag that displays security warnings. +- A 12-character minimum password length is enforced, following [NIST SP 800-63B][nist-sp800-63b] guidance. +- SCRAM credentials are generated with 10,000 iterations (above the RFC-5802 minimum of 4,096) and 20 bytes of random salt. + +### Configuration model + +```yaml +filters: + - type: SaslTermination + config: + mechanisms: + SCRAM-SHA-256: + credentialStore: KeystoreScramCredentialStoreService + credentialStoreConfig: + file: /path/to/credentials.p12 + storePassword: + file: /etc/kroxylicious/keystore-password.txt + storeType: PKCS12 + OAUTHBEARER: + jwksEndpointUrl: https://idp.example.com/.well-known/jwks.json + expectedAudience: kafka + expectedIssuer: https://idp.example.com +``` + +The `mechanisms` map is keyed by IANA-registered mechanism name. The config shape for each entry depends on the mechanism: SCRAM mechanisms use `credentialStore`/`credentialStoreConfig`, while OAUTHBEARER uses JWKS endpoint configuration directly. + +### Module architecture + +The implementation is organized into three modules, following the same pattern as the existing KMS modules (`kroxylicious-kms`, `kroxylicious-kms-provider-*`): + +| Module | Purpose | +|--------|---------| +| `kroxylicious-filters/kroxylicious-sasl-termination` | The termination filter, state machine, mechanism handler extensibility, and all built-in mechanism implementations | +| `kroxylicious-sasl-credential-store` | Public API: defines the credential store SPI used by SCRAM mechanism handlers | +| `kroxylicious-sasl-credential-store-providers/kroxylicious-sasl-credential-store-provider-keystore` | First-party SCRAM credential provider: Java KeyStore-backed implementation with CLI management tool | + +## Security model + +### Credential storage + +- **KeyStore encryption:** Credentials are stored in Java KeyStore files, encrypted with the KeyStore password. File-system permissions and KeyStore passwords are the primary access controls. +- **PasswordProvider abstraction:** Production deployments should use file-based passwords rather than inline passwords in configuration. The `PasswordProvider` interface supports both. +- **In-memory handling:** `ScramCredential` uses defensive copies for all `byte[]` fields (salt, serverKey, storedKey) in both the constructor and accessors, preventing callers from mutating stored credential data. `toString()` redacts sensitive fields. + +### SCRAM protocol correctness + +The implementation delegates to Kafka's own `SaslServer` for SCRAM, which is widely deployed and well-tested. The handler is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. + +### Username enumeration prevention + +When a user is not found in the credential store, the handler returns a generic `"Authentication failed"` error message, identical to the message returned for incorrect credentials. The error does not reveal whether the username exists. + +### Timing side-channel + +**Known limitation:** When a user is not found, the handler returns failure immediately (after the async credential lookup completes). When a user _is_ found, the handler creates a `SaslServer`, processes the first SCRAM message, and returns a challenge. An attacker measuring response times could distinguish these two paths. + +**Mitigation recommendation:** For non-existent users, generate a deterministic fake credential using a keyed hash of the username as the salt, and continue the SCRAM exchange as if the user existed. The authentication will fail at the proof verification stage, but the timing will be indistinguishable from a real user with incorrect credentials. This is the approach used by some SCRAM implementations and should be added as a future improvement. + +### Connection lifecycle safety + +- The sealed state machine prevents invalid state transitions at compile time. +- The security barrier is enforced for all non-SASL request types. Unauthenticated requests are rejected and the connection is closed. +- On authentication failure, the connection is closed immediately. + +### CLI tool security + +- Interactive password prompts prevent exposure of passwords in shell history and process listings. +- The `--unlock-insecure-options` flag gates command-line password arguments with explicit security warnings. +- 12-character minimum password length follows NIST SP 800-63B recommendations. + +### Code quality findings + +Two issues identified during security review: + +1. **Blocking call on event loop:** `SaslTerminationFilter.handleAuthenticationFailure()` calls `.toCompletableFuture().join()` on a future that should already be complete. While functionally correct, this violates the project's performance rules ("Never call `.join()` or `.get()` on futures in filter code") and should be refactored to fully async handling. + +2. **Logging convention violation:** `ScramHandler.evaluateResponse()` uses `addArgument()` for message interpolation, violating the project's logging convention which requires `addKeyValue()` for structured logging. + +Both issues should be fixed before merge. + +### Threats considered but out of scope + +- **Compromised KeyStore files:** Protecting the KeyStore file at rest is an operational concern (file permissions, encryption at rest) rather than an application concern. +- **SCRAM channel binding:** [RFC 5802 Section 6][rfc5802-s6] describes channel binding for SCRAM. Kafka does not use SCRAM channel binding, so this implementation follows Kafka's approach. +- **Reauthentication (KIP-368):** The current implementation does not support SASL reauthentication. This is acceptable for the initial release and can be added later. + +## Affected/not affected projects + +**New modules:** +- `kroxylicious-sasl-credential-store` — public API module +- `kroxylicious-sasl-credential-store-providers/kroxylicious-sasl-credential-store-provider-keystore` — KeyStore provider +- `kroxylicious-filters/kroxylicious-sasl-termination` — termination filter + +**Modified modules:** +- `kroxylicious-bom` — new dependency declarations +- Root `pom.xml` — new module entries +- `kroxylicious-integration-tests` — SASL termination integration tests +- `kroxylicious-docs` — authentication guide (expanded from SASL inspection guide) + +**Not affected:** +- `kroxylicious-api` — no API changes needed (uses existing `clientSaslAuthenticationSuccess`/`clientSaslAuthenticationFailure` from proposal 006) +- `kroxylicious-runtime` — no runtime changes +- `kroxylicious-kms` and KMS providers — unrelated +- `kroxylicious-kubernetes` — no operator changes (the termination filter is configured via standard filter configuration) + +## Compatibility + +This is a new feature with no breaking changes: +- Existing proxy configurations continue to work unchanged. +- The SASL inspection filter is unaffected and can still be used for passthrough inspection. +- The OAUTHBEARER validation filter is unaffected. +- The credential store API (`kroxylicious-sasl-credential-store`) is a new public API. Once released, it will follow the project's API stability rules. + +## Rejected alternatives + +### Generic CredentialStore covering all mechanisms + +A single `CredentialStore` interface serving both SCRAM and OAUTHBEARER was considered. This was rejected because: +- SCRAM uses stored credential lookup (username → salt, iterations, server key, stored key). +- OAUTHBEARER uses token validation against a JWKS endpoint (no stored credentials at all). +- A generic interface would either be too abstract to be useful or would leak mechanism-specific concepts into the abstraction. + +Instead, each mechanism family manages its own resources. The `MechanismHandlerFactory` is the point where mechanism-specific resources (credential stores, JWKS handlers) are injected. + +### Using @Plugin for mechanism handlers + +Making `MechanismHandlerFactory` a user-facing plugin (with `@Plugin` annotation and plugin discovery) was considered. This was rejected because: +- Mechanism handlers are internal implementation details, not user-facing extension points. +- Users configure _mechanisms_, not _handlers_. The mapping from mechanism name to handler is an implementation concern. +- `ServiceLoader` discovery is sufficient for internal extensibility. + +### Extending the OAUTHBEARER validation filter + +Adding SASL termination support to the existing OAUTHBEARER validation filter was considered. This was rejected because: +- The validation filter performs a fundamentally different operation: it validates tokens then _forwards_ the SASL exchange to the broker. Termination _short-circuits_ — the broker never sees SASL traffic. +- The filter lifecycle, state management, and security barrier requirements are different. +- Combining both would create a complex filter with two distinct operational modes. + +### PLAIN mechanism support + +Supporting SASL PLAIN was deferred because: +- PLAIN transmits passwords in cleartext (Base64 encoded, not encrypted), making it unsuitable for production use without TLS. +- SCRAM provides mutual authentication and never transmits the password (though should also be used with TLS to avoid MitM attacks). +- If PLAIN support is needed in the future, it could be added as a new `MechanismHandler` implementation. + +## References + +- [Proposal 004 — Terminology for Authentication][proposal-004] +- [Proposal 006 — API to expose client SASL information to Filters][proposal-006] +- [RFC 4422 — Simple Authentication and Security Layer (SASL)][rfc4422] +- [RFC 5802 — Salted Challenge Response Authentication Mechanism (SCRAM)][rfc5802] +- [RFC 6750 — The OAuth 2.0 Authorization Framework: Bearer Token Usage][rfc6750] +- [RFC 7628 — A Set of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth][rfc7628] +- [KIP-84 — Support SASL SCRAM mechanisms][kip84] +- [KIP-255 — OAuth Authentication via SASL/OAUTHBEARER][kip255] +- [NIST SP 800-63B — Digital Identity Guidelines: Authentication and Lifecycle Management][nist-sp800-63b] + +[proposal-004]: 004-terminology-for-authentication.md +[proposal-006]: 006-filter-api-to-expose-client-sasl-info.md +[proposal-072]: 070-routing-api.md +[rfc4422]: https://www.rfc-editor.org/rfc/rfc4422 +[rfc5802]: https://www.rfc-editor.org/rfc/rfc5802 +[rfc5802-s6]: https://www.rfc-editor.org/rfc/rfc5802#section-6 +[rfc6750]: https://www.rfc-editor.org/rfc/rfc6750 +[rfc7628]: https://www.rfc-editor.org/rfc/rfc7628 +[kip84]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms +[kip255]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75968876 +[nist-sp800-63b]: https://pages.nist.gov/800-63-4/sp800-63b.html +[sasl-inspection]: https://kroxylicious.io/kroxylicious/#assembly-sasl-inspection +[oauthbearer-validation]: https://kroxylicious.io/kroxylicious/#assembly-configuring-oauth-bearer-validation-filter From d9f34f95eec21221186563b37d5ad9162fb0a124 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 00:39:53 +0000 Subject: [PATCH 02/52] docs(proposal): add KIP-368 reauthentication to SASL termination proposal Adds reauthentication section covering connectionsMaxReauth config, session lifetime computation, server-side expiry enforcement, and OAUTHBEARER token lifetime extraction. Removes reauthentication from out-of-scope section. Updates state diagram and config example. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index 8443ab6b..27d98d05 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -68,18 +68,34 @@ RequiringHandshake ──→ RequiringAuthenticate ←──╮ │ │ ├─ (multi-round) ──╯ │ - ├──→ Authenticated (terminal) + ├──→ Authenticated ──→ (reauth) ──→ RequiringAuthenticate + │ │ + │ └──→ (expired + non-SASL request) ──→ reject & close │ └──→ Failed (terminal) ``` - **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests, which negotiate the mechanism and transition to `RequiringAuthenticate`. - **RequiringAuthenticate:** Accepts `SASL_AUTHENTICATE` requests. Loops back to itself for multi-round mechanisms (e.g. SCRAM). Carries a reference to the `MechanismHandler` for the negotiated mechanism. -- **Authenticated:** Terminal success state. The filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters, then forwards all subsequent requests. +- **Authenticated:** Success state. The filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters, then forwards all subsequent requests. If reauthentication is configured, this state also stores the session expiry time and allows transition back to `RequiringAuthenticate` via a new `SASL_HANDSHAKE`. - **Failed:** Terminal failure state. The connection is closed. The sealed interface prevents creation of invalid states at compile time. +#### Reauthentication (KIP-368) + +The filter supports [KIP-368][kip368] reauthentication. When `connectionsMaxReauth` is configured, the filter includes a `sessionLifetimeMs` value in the `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. + +**Session lifetime computation:** The effective session lifetime is the minimum of: +1. The configured `connectionsMaxReauth` value. +2. The handler-reported credential/token lifetime (e.g. the JWT token's expiry for OAUTHBEARER). + +If either value is zero (no opinion / no expiry), the other is used. If both are zero, no reauthentication is required. + +**Client behaviour:** Standard Kafka clients (4.0+) handle reauthentication transparently via the `Selector`. When the session nears expiry, the client sends a new `SASL_HANDSHAKE` + `SASL_AUTHENTICATE` sequence over the existing connection. This is invisible to application code. + +**Server-side enforcement:** If the session has expired and a non-SASL request arrives, the filter rejects it with `SASL_AUTHENTICATION_FAILED` and closes the connection. `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests are always accepted regardless of session expiry, to allow reauthentication. + ### Mechanism handler extension point The filter delegates the actual authentication exchange to mechanism-specific handlers, discovered via an internal extension point: @@ -110,7 +126,7 @@ The OAUTHBEARER handler validates JWT bearer tokens at the proxy without forward The handler uses Kafka's `OAuthBearerValidatorCallbackHandler` for JWT validation, the same mechanism used by the existing OAUTHBEARER validation filter. The `OauthBearerHandlerFactory` manages the JWKS endpoint configuration and callback handler lifecycle: at `initialize()`-time it configures the callback handler with the JWKS endpoint, expected audience/issuer, and refresh settings; per-connection handlers receive the shared callback handler and use it to create a `SaslServer` via the JSSE/SASL framework. -OAUTHBEARER is architecturally the simpler mechanism — it requires no credential store. The factory's only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). +OAUTHBEARER is architecturally the simpler mechanism — it requires no credential store. The factory's only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). After successful authentication, the handler extracts the token's remaining lifetime from the `SaslServer`'s negotiated `CREDENTIAL.LIFETIME.MS` property for use in session lifetime computation (see [Reauthentication](#reauthentication-kip-368)). **Key differences from the existing OAUTHBEARER validation filter:** - The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, the termination handler validates tokens and _short-circuits_ — the broker never sees a SASL exchange. @@ -177,6 +193,7 @@ Security measures: filters: - type: SaslTermination config: + connectionsMaxReauth: 1h mechanisms: SCRAM-SHA-256: credentialStore: KeystoreScramCredentialStoreService @@ -193,6 +210,8 @@ filters: The `mechanisms` map is keyed by IANA-registered mechanism name. The config shape for each entry depends on the mechanism: SCRAM mechanisms use `credentialStore`/`credentialStoreConfig`, while OAUTHBEARER uses JWKS endpoint configuration directly. +The optional `connectionsMaxReauth` sets the maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. + ### Module architecture The implementation is organized into three modules, following the same pattern as the existing KMS modules (`kroxylicious-kms`, `kroxylicious-kms-provider-*`): @@ -251,7 +270,6 @@ Both issues should be fixed before merge. - **Compromised KeyStore files:** Protecting the KeyStore file at rest is an operational concern (file permissions, encryption at rest) rather than an application concern. - **SCRAM channel binding:** [RFC 5802 Section 6][rfc5802-s6] describes channel binding for SCRAM. Kafka does not use SCRAM channel binding, so this implementation follows Kafka's approach. -- **Reauthentication (KIP-368):** The current implementation does not support SASL reauthentication. This is acceptable for the initial release and can be added later. ## Affected/not affected projects @@ -322,6 +340,7 @@ Supporting SASL PLAIN was deferred because: - [RFC 7628 — A Set of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth][rfc7628] - [KIP-84 — Support SASL SCRAM mechanisms][kip84] - [KIP-255 — OAuth Authentication via SASL/OAUTHBEARER][kip255] +- [KIP-368 — Allow SASL Connections to Periodically Re-Authenticate][kip368] - [NIST SP 800-63B — Digital Identity Guidelines: Authentication and Lifecycle Management][nist-sp800-63b] [proposal-004]: 004-terminology-for-authentication.md @@ -334,6 +353,7 @@ Supporting SASL PLAIN was deferred because: [rfc7628]: https://www.rfc-editor.org/rfc/rfc7628 [kip84]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms [kip255]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75968876 +[kip368]: https://cwiki.apache.org/confluence/spaces/KAFKA/pages/89068981/KIP-368+Allow+SASL+Connections+to+Periodically+Re-Authenticate [nist-sp800-63b]: https://pages.nist.gov/800-63-4/sp800-63b.html [sasl-inspection]: https://kroxylicious.io/kroxylicious/#assembly-sasl-inspection [oauthbearer-validation]: https://kroxylicious.io/kroxylicious/#assembly-configuring-oauth-bearer-validation-filter From 49aede467ce8d879eac1e0cc9d5c8dd0a44fcb6f Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 04:27:15 +0000 Subject: [PATCH 03/52] docs(proposal): document timing side-channel mitigation Replaces the "known limitation / future improvement" text with documentation of the implemented fixed-delay mitigation. Removes stale code quality findings that were already fixed. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index 27d98d05..ad59fcbc 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -238,11 +238,9 @@ The implementation delegates to Kafka's own `SaslServer` for SCRAM, which is wid When a user is not found in the credential store, the handler returns a generic `"Authentication failed"` error message, identical to the message returned for incorrect credentials. The error does not reveal whether the username exists. -### Timing side-channel +### Timing side-channel mitigation -**Known limitation:** When a user is not found, the handler returns failure immediately (after the async credential lookup completes). When a user _is_ found, the handler creates a `SaslServer`, processes the first SCRAM message, and returns a challenge. An attacker measuring response times could distinguish these two paths. - -**Mitigation recommendation:** For non-existent users, generate a deterministic fake credential using a keyed hash of the username as the salt, and continue the SCRAM exchange as if the user existed. The authentication will fail at the proof verification stage, but the timing will be indistinguishable from a real user with incorrect credentials. This is the approach used by some SCRAM implementations and should be added as a future improvement. +Without mitigation, an attacker could distinguish existing from non-existing users by measuring response times: credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists. Rather than trying to equalize these inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the SCRAM handler applies a fixed delay to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. ### Connection lifecycle safety @@ -256,16 +254,6 @@ When a user is not found in the credential store, the handler returns a generic - The `--unlock-insecure-options` flag gates command-line password arguments with explicit security warnings. - 12-character minimum password length follows NIST SP 800-63B recommendations. -### Code quality findings - -Two issues identified during security review: - -1. **Blocking call on event loop:** `SaslTerminationFilter.handleAuthenticationFailure()` calls `.toCompletableFuture().join()` on a future that should already be complete. While functionally correct, this violates the project's performance rules ("Never call `.join()` or `.get()` on futures in filter code") and should be refactored to fully async handling. - -2. **Logging convention violation:** `ScramHandler.evaluateResponse()` uses `addArgument()` for message interpolation, violating the project's logging convention which requires `addKeyValue()` for structured logging. - -Both issues should be fixed before merge. - ### Threats considered but out of scope - **Compromised KeyStore files:** Protecting the KeyStore file at rest is an operational concern (file permissions, encryption at rest) rather than an application concern. From 0df6a4a6d52b837ecc6b2ed435b1eb55b05971e8 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 04:50:15 +0000 Subject: [PATCH 04/52] docs(proposal): document keystore file permission enforcement Adds file permission check to credential storage security section. Removes compromised keystore files from out-of-scope threats. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index ad59fcbc..f5865fce 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -228,6 +228,7 @@ The implementation is organized into three modules, following the same pattern a - **KeyStore encryption:** Credentials are stored in Java KeyStore files, encrypted with the KeyStore password. File-system permissions and KeyStore passwords are the primary access controls. - **PasswordProvider abstraction:** Production deployments should use file-based passwords rather than inline passwords in configuration. The `PasswordProvider` interface supports both. +- **File permission enforcement:** On POSIX systems, the credential store refuses to load a KeyStore file that has group or world read/write permissions. This prevents accidental exposure of credential material through overly permissive file modes. - **In-memory handling:** `ScramCredential` uses defensive copies for all `byte[]` fields (salt, serverKey, storedKey) in both the constructor and accessors, preventing callers from mutating stored credential data. `toString()` redacts sensitive fields. ### SCRAM protocol correctness @@ -256,7 +257,6 @@ Without mitigation, an attacker could distinguish existing from non-existing use ### Threats considered but out of scope -- **Compromised KeyStore files:** Protecting the KeyStore file at rest is an operational concern (file permissions, encryption at rest) rather than an application concern. - **SCRAM channel binding:** [RFC 5802 Section 6][rfc5802-s6] describes channel binding for SCRAM. Kafka does not use SCRAM channel binding, so this implementation follows Kafka's approach. ## Affected/not affected projects From 05d58236fb26cd3befa228eb64db603d516bec88 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 05:57:04 +0000 Subject: [PATCH 05/52] docs(proposal): document Kafka internal API dependencies Lists all non-public Kafka APIs used by the implementation, notes that the SPI types are clean of Kafka dependencies, and flags that KeystoreCredentialManager exposes ScramMechanism in its method signatures. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index f5865fce..c55804b6 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -286,6 +286,24 @@ This is a new feature with no breaking changes: - The OAUTHBEARER validation filter is unaffected. - The credential store API (`kroxylicious-sasl-credential-store`) is a new public API. Once released, it will follow the project's API stability rules. +## Kafka internal API dependencies + +This implementation uses several Kafka APIs that are not part of the [published Kafka javadoc][kafka-javadoc] and may change without notice in future Kafka releases. + +**`org.apache.kafka.common.message.*` and `org.apache.kafka.common.protocol.*`** (`ApiKeys`, `Errors`, `RequestHeaderData`, `SaslAuthenticateRequestData`, etc.) — These are the Kafka protocol message classes. They are not in Kafka's public javadoc, but they are a foundational dependency for Kroxylicious: the filter API itself (`RequestFilter`) exposes these types. All Kroxylicious filters depend on them. + +**`org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServerProvider`** — Called once (`initialize()`) to register the OAUTHBEARER SASL mechanism with the JVM's security provider infrastructure. The existing OAUTHBEARER validation filter uses this in the same way. There is no public API alternative. + +**`org.apache.kafka.common.security.scram.internals.ScramMechanism`** — An enum identifying SCRAM-SHA-256 and SCRAM-SHA-512. Used internally by the SCRAM handler factories and the keystore credential manager. There is no public API equivalent. + +**`org.apache.kafka.common.security.scram.internals.ScramFormatter`** — Used by `KeystoreCredentialManager` to derive salted passwords, server keys, and stored keys from plaintext passwords. This is the only implementation of SCRAM key derivation available in the Kafka client library. There is no public API equivalent. + +All of these dependencies are contained within the implementation modules. The public SPI types (`ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, `MechanismHandler`, `MechanismHandlerFactory`, `AuthenticationResult`) do not reference any Kafka types. Implementors of the credential store SPI are not transitively exposed to Kafka internal APIs. + +The `KeystoreCredentialManager` class does expose `ScramMechanism` in its public method signatures (`addUser`, `updatePassword`, `generateKeyStore`, `generateScramCredential`). This class is in the provider module, not the SPI, so it is not part of the formal public API contract — but external code that uses the credential manager directly would take a dependency on this internal Kafka type. + +[kafka-javadoc]: https://kafka.apache.org/43/javadoc/index.html + ## Rejected alternatives ### Generic CredentialStore covering all mechanisms From 0e086f433c2a14ca91955ecc27a2dcc0a2f1c9e5 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 06:07:56 +0000 Subject: [PATCH 06/52] docs(proposal): reference Proposal 116 for Kafka internal API dependencies Each Kafka internal API dependency now notes how Proposal 116 (Kafka API migration) would address it: protocol/message classes would be fully owned; SASL security classes would gain namespace stability but remain functional implementations with ongoing maintenance burden. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index c55804b6..9faffe9a 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -288,21 +288,22 @@ This is a new feature with no breaking changes: ## Kafka internal API dependencies -This implementation uses several Kafka APIs that are not part of the [published Kafka javadoc][kafka-javadoc] and may change without notice in future Kafka releases. +This implementation uses several Kafka APIs that are not part of the [published Kafka javadoc][kafka-javadoc] and may change without notice in future Kafka releases. [Proposal 116][proposal-116] (Kafka API migration) would bring all of these under a Kroxylicious-owned namespace, insulating this code from upstream Kafka reorganisations. -**`org.apache.kafka.common.message.*` and `org.apache.kafka.common.protocol.*`** (`ApiKeys`, `Errors`, `RequestHeaderData`, `SaslAuthenticateRequestData`, etc.) — These are the Kafka protocol message classes. They are not in Kafka's public javadoc, but they are a foundational dependency for Kroxylicious: the filter API itself (`RequestFilter`) exposes these types. All Kroxylicious filters depend on them. +**`org.apache.kafka.common.message.*` and `org.apache.kafka.common.protocol.*`** (`ApiKeys`, `Errors`, `RequestHeaderData`, `SaslAuthenticateRequestData`, etc.) — The Kafka protocol message classes. Not in Kafka's public javadoc, but a foundational dependency for Kroxylicious: the filter API itself (`RequestFilter`) exposes these types. All Kroxylicious filters depend on them. These are the primary target of [Proposal 116][proposal-116] and would become fully Kroxylicious-owned. -**`org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServerProvider`** — Called once (`initialize()`) to register the OAUTHBEARER SASL mechanism with the JVM's security provider infrastructure. The existing OAUTHBEARER validation filter uses this in the same way. There is no public API alternative. +**`org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServerProvider`** — Called once (`initialize()`) to register the OAUTHBEARER SASL mechanism with the JVM's security provider infrastructure. The existing OAUTHBEARER validation filter uses this in the same way. There is no public API alternative. [Proposal 116][proposal-116] would copy this into the Kroxylicious namespace, giving stability control, but the functional dependency on Kafka's JSSE provider registration code remains. -**`org.apache.kafka.common.security.scram.internals.ScramMechanism`** — An enum identifying SCRAM-SHA-256 and SCRAM-SHA-512. Used internally by the SCRAM handler factories and the keystore credential manager. There is no public API equivalent. +**`org.apache.kafka.common.security.scram.internals.ScramMechanism`** — An enum identifying SCRAM-SHA-256 and SCRAM-SHA-512. Used internally by the SCRAM handler factories and the keystore credential manager. There is no public API equivalent. [Proposal 116][proposal-116] would own this type, but it is a trivial enum that could equally be replaced with a Kroxylicious-native type. -**`org.apache.kafka.common.security.scram.internals.ScramFormatter`** — Used by `KeystoreCredentialManager` to derive salted passwords, server keys, and stored keys from plaintext passwords. This is the only implementation of SCRAM key derivation available in the Kafka client library. There is no public API equivalent. +**`org.apache.kafka.common.security.scram.internals.ScramFormatter`** — Used by `KeystoreCredentialManager` to derive salted passwords, server keys, and stored keys from plaintext passwords. This is the only implementation of SCRAM key derivation available in the Kafka client library. There is no public API equivalent. [Proposal 116][proposal-116] would copy this into the Kroxylicious namespace, but unlike the protocol data classes, `ScramFormatter` is a functional security implementation (PBKDF2, HMAC) — the maintenance burden of keeping it current remains. All of these dependencies are contained within the implementation modules. The public SPI types (`ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, `MechanismHandler`, `MechanismHandlerFactory`, `AuthenticationResult`) do not reference any Kafka types. Implementors of the credential store SPI are not transitively exposed to Kafka internal APIs. The `KeystoreCredentialManager` class does expose `ScramMechanism` in its public method signatures (`addUser`, `updatePassword`, `generateKeyStore`, `generateScramCredential`). This class is in the provider module, not the SPI, so it is not part of the formal public API contract — but external code that uses the credential manager directly would take a dependency on this internal Kafka type. [kafka-javadoc]: https://kafka.apache.org/43/javadoc/index.html +[proposal-116]: https://github.com/kroxylicious/design/pull/116 ## Rejected alternatives From f93c08e13dca384b5b55166055fba1777a66d243 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 22:07:57 +0000 Subject: [PATCH 07/52] docs(proposal): rename connectionsMaxReauth to maxTimeBeforeReauth Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index 9faffe9a..447649ad 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -6,7 +6,7 @@ SASL termination allows the Kroxylicious proxy to authenticate Kafka clients dir Kroxylicious currently handles client SASL authentication in a number of ways: -1. **SASL Passthrough**: The proxy forwards SASL exchanges unmodified between client and broker. The broker performs all authentication. +1. **SASL Passthrough**: The proxy forwards SASL exchanges unmodified between client and broker. The broker performs all authentication. The proxy remains ignorant of the client subject. 2. **SASL Passthrough Inspection**: The [SASL inspection filter][sasl-inspection] observes SASL exchanges as they pass through, extracting the client's authorization ID without making authentication decisions itself. This supports SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, and PLAIN mechanisms. @@ -32,7 +32,7 @@ With SASL termination, the proxy authenticates clients using credentials stored ### Authentication protocol translation -The proxy can authenticate clients using one SASL mechanism (e.g. SCRAM-SHA-256) while using an entirely different authentication mechanism to connect to the broker (e.g. mTLS, or a service account). This enables: +The proxy can authenticate clients using one SASL mechanism (e.g. `SCRAM-SHA-256`) while using an entirely different authentication mechanism to connect to the broker (e.g. mTLS, or `OAUTHBEARER`). This enables: - Migrating broker authentication without changing client configurations. - Using client-friendly mechanisms even when the broker supports only a limited set. @@ -53,7 +53,7 @@ A key problem with any passthrough-based technique is that it depends on the ava ## Proposal This proposal aims to support for the following SASL mechanisms: `SCRAM-SHA-256`, `SCRAM-SHA-512` and `OAUTHBEARER`. -It also aims to be flexible, so as to allow other mechanisms to be supported either in the future, or as plugins. +It also aims to be flexible, so as to allow other mechanisms to be supported either in the future. ### The filter @@ -64,9 +64,9 @@ The SASL termination filter intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` The filter maintains per-connection state using a sealed interface `State` with four concrete states: ``` -RequiringHandshake ──→ RequiringAuthenticate ←──╮ - │ │ - ├─ (multi-round) ──╯ +RequiringHandshake ──→ RequiringAuthenticate ←────╮ + │ │ + ├─→ (multi-round) ──╯ │ ├──→ Authenticated ──→ (reauth) ──→ RequiringAuthenticate │ │ @@ -84,10 +84,10 @@ The sealed interface prevents creation of invalid states at compile time. #### Reauthentication (KIP-368) -The filter supports [KIP-368][kip368] reauthentication. When `connectionsMaxReauth` is configured, the filter includes a `sessionLifetimeMs` value in the `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. +The filter supports [KIP-368][kip368] reauthentication. When `maxTimeBeforeReauth` is configured, the filter includes a `sessionLifetimeMs` value in the `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. **Session lifetime computation:** The effective session lifetime is the minimum of: -1. The configured `connectionsMaxReauth` value. +1. The configured `maxTimeBeforeReauth` value. 2. The handler-reported credential/token lifetime (e.g. the JWT token's expiry for OAUTHBEARER). If either value is zero (no opinion / no expiry), the other is used. If both are zero, no reauthentication is required. @@ -193,7 +193,7 @@ Security measures: filters: - type: SaslTermination config: - connectionsMaxReauth: 1h + maxTimeBeforeReauth: 1h mechanisms: SCRAM-SHA-256: credentialStore: KeystoreScramCredentialStoreService @@ -210,7 +210,7 @@ filters: The `mechanisms` map is keyed by IANA-registered mechanism name. The config shape for each entry depends on the mechanism: SCRAM mechanisms use `credentialStore`/`credentialStoreConfig`, while OAUTHBEARER uses JWKS endpoint configuration directly. -The optional `connectionsMaxReauth` sets the maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. +The optional `maxTimeBeforeReauth` sets the maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. ### Module architecture From b15988d7ef8278464463b315ea1360d58ea9e640 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 22:26:23 +0000 Subject: [PATCH 08/52] docs(proposal): require audience/issuer and document OAUTHBEARER known gaps expectedAudience and expectedIssuer are now required fields. Documents three known limitations: no TLS config for JWKS endpoint, no rate limiting, and hardcoded JWT validator class. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index 447649ad..ef43a76c 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -132,6 +132,13 @@ OAUTHBEARER is architecturally the simpler mechanism — it requires no credenti - The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, the termination handler validates tokens and _short-circuits_ — the broker never sees a SASL exchange. - The handler factory owns its callback handler and JWKS configuration, receiving them at `initialize()`-time rather than requiring a credential store. +**Security requirements:** The `expectedAudience` and `expectedIssuer` fields are required. Without audience validation, a token issued for a different service would be accepted; without issuer validation, tokens from any issuer whose keys happen to be in the JWKS would be accepted. + +**Known limitations:** +- **TLS configuration for the JWKS endpoint:** Kafka's `OAuthBearerValidatorCallbackHandler` uses an internal HTTP client with no TLS configuration surface. There is currently no way to configure custom trust stores or client certificates for HTTPS communication with the JWKS endpoint. The JVM's default trust store is used. This is a limitation inherited from Kafka's callback handler and shared with the existing OAUTHBEARER validation filter. +- **Rate limiting:** The handler does not implement rate limiting or brute-force protection for failed authentication attempts. The existing OAUTHBEARER validation filter has Caffeine-based rate limiting with exponential backoff that could serve as a reference for a future implementation. +- **Custom JWT validator:** The handler hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. + ### SCRAM implementation SCRAM is more complex than OAUTHBEARER because it is a multi-round challenge-response protocol that requires stored credentials. From fef813e12fa605c9bd50673b9f6a3b11498b1c4a Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 22:56:26 +0000 Subject: [PATCH 09/52] docs(proposal): restructure proposal into per-component documentation Reorganizes the Proposal section into 7 components, each with: - Summary and key features - API surfaces with actual Java interface code - Configuration tables with optionality and defaults - Threats and mitigations tables - Known limitations Components: SaslTermination filter, MechanismHandler extension point, SCRAM handler, OAUTHBEARER handler, ScramCredentialStore SPI, KeyStore provider, module architecture. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 514 ++++++++++++++++++++++++------ 1 file changed, 411 insertions(+), 103 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index ef43a76c..f0d1e040 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -52,27 +52,38 @@ A key problem with any passthrough-based technique is that it depends on the ava ## Proposal -This proposal aims to support for the following SASL mechanisms: `SCRAM-SHA-256`, `SCRAM-SHA-512` and `OAUTHBEARER`. -It also aims to be flexible, so as to allow other mechanisms to be supported either in the future. +This proposal aims to support the following SASL mechanisms: `SCRAM-SHA-256`, `SCRAM-SHA-512` and `OAUTHBEARER`. +It also aims to be flexible, so as to allow other mechanisms to be supported in the future. -### The filter +The proposal is organized per-component. Each component section covers its summary, API surfaces, configuration, threats and mitigations, and known limitations. -The SASL termination filter intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, authenticating clients at the proxy and short-circuiting the responses without forwarding them to the broker. It enforces a security barrier: until a client has successfully authenticated, the only requests permitted are `API_VERSIONS`, `SASL_HANDSHAKE`, and `SASL_AUTHENTICATE`. All other request types are rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. +### Component 1: SaslTermination filter + +#### Summary + +The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, authenticating clients at the proxy and short-circuiting the responses without forwarding them to the broker. + +Key features: + +- **Security barrier.** Until a client has successfully authenticated, the only requests permitted are `API_VERSIONS`, `SASL_HANDSHAKE`, and `SASL_AUTHENTICATE`. All other request types are rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. +- **State machine.** Per-connection authentication state is modelled as a sealed interface with four concrete states, preventing invalid transitions at compile time. +- **Reauthentication (KIP-368).** When `maxTimeBeforeReauth` is configured, the filter includes a `sessionLifetimeMs` value in `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. Sessions that expire without reauthentication are rejected and closed. +- **Mechanism dispatch.** The filter delegates each authentication exchange to a `MechanismHandler` obtained from the appropriate `MechanismHandlerFactory` (see Component 2). The filter itself is mechanism-agnostic. #### State machine The filter maintains per-connection state using a sealed interface `State` with four concrete states: ``` -RequiringHandshake ──→ RequiringAuthenticate ←────╮ - │ │ - ├─→ (multi-round) ──╯ - │ - ├──→ Authenticated ──→ (reauth) ──→ RequiringAuthenticate - │ │ - │ └──→ (expired + non-SASL request) ──→ reject & close - │ - └──→ Failed (terminal) +RequiringHandshake ──> RequiringAuthenticate <────╮ + | | + |─> (multi-round) ──╯ + | + |──> Authenticated ──> (reauth) ──> RequiringAuthenticate + | | + | └──> (expired + non-SASL request) ──> reject & close + | + └──> Failed (terminal) ``` - **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests, which negotiate the mechanism and transition to `RequiringAuthenticate`. @@ -80,11 +91,9 @@ RequiringHandshake ──→ RequiringAuthenticate ←────╮ - **Authenticated:** Success state. The filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters, then forwards all subsequent requests. If reauthentication is configured, this state also stores the session expiry time and allows transition back to `RequiringAuthenticate` via a new `SASL_HANDSHAKE`. - **Failed:** Terminal failure state. The connection is closed. -The sealed interface prevents creation of invalid states at compile time. - #### Reauthentication (KIP-368) -The filter supports [KIP-368][kip368] reauthentication. When `maxTimeBeforeReauth` is configured, the filter includes a `sessionLifetimeMs` value in the `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. +The filter supports [KIP-368][kip368] reauthentication. **Session lifetime computation:** The effective session lifetime is the minimum of: 1. The configured `maxTimeBeforeReauth` value. @@ -96,138 +105,437 @@ If either value is zero (no opinion / no expiry), the other is used. If both are **Server-side enforcement:** If the session has expired and a non-SASL request arrives, the filter rejects it with `SASL_AUTHENTICATION_FAILED` and closes the connection. `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests are always accepted regardless of session expiry, to allow reauthentication. -### Mechanism handler extension point +#### API surfaces + +The filter is a standard Kroxylicious `FilterFactory` plugin. It does not define any new public API. It uses: + +- `FilterFactory` (from `kroxylicious-api`) -- the standard filter factory contract. +- `RequestFilter` (from `kroxylicious-api`) -- for intercepting requests. +- `FilterContext.clientSaslAuthenticationSuccess()` / `clientSaslAuthenticationFailure()` (from `kroxylicious-api`, added by Proposal 006) -- to propagate authentication outcomes. +- `MechanismHandlerFactory` (internal, see Component 2) -- for mechanism dispatch. + +#### Configuration + +The filter is configured via `SaslTerminationConfig`: + +| Option | Type | Required | Default | Description | +|--------|------|----------|---------|-------------| +| `mechanisms` | `Map` | Yes | -- | Map of IANA-registered mechanism name to mechanism-specific configuration. At least one entry is required. | +| `maxTimeBeforeReauth` | `Duration` | No | disabled | Maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. | + +The `mechanisms` map values are polymorphic. Jackson deduction-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION)`) resolves the concrete type from the fields present: + +- If the entry contains `credentialStore` and `credentialStoreConfig`, it deserializes as `ScramMechanismConfig`. +- If the entry contains `jwksEndpointUrl`, `expectedAudience`, and `expectedIssuer`, it deserializes as `OauthBearerMechanismConfig`. + +This means the mechanism map key (e.g. `SCRAM-SHA-256`) selects which `MechanismHandlerFactory` handles the exchange, while the value's field structure determines which config type Jackson produces. There is no explicit type discriminator field. + +**Example configuration:** + +```yaml +filters: + - type: SaslTermination + config: + maxTimeBeforeReauth: 1h + mechanisms: + SCRAM-SHA-256: + credentialStore: KeystoreScramCredentialStoreService + credentialStoreConfig: + file: /path/to/credentials.p12 + storePassword: + file: /etc/kroxylicious/keystore-password.txt + storeType: PKCS12 + OAUTHBEARER: + jwksEndpointUrl: https://idp.example.com/.well-known/jwks.json + expectedAudience: kafka + expectedIssuer: https://idp.example.com +``` + +#### Threats and mitigations + +| Threat | Mitigation | +|--------|------------| +| Unauthenticated request bypass -- a client sends Kafka protocol requests (Produce, Fetch, etc.) before completing SASL authentication. | The security barrier rejects all non-SASL request types until the state reaches `Authenticated`. Rejected requests receive `SASL_AUTHENTICATION_FAILED` and the connection is closed immediately. | +| Session expiry evasion -- an authenticated client continues sending requests after its session has expired without reauthenticating. | On every non-SASL request in the `Authenticated` state, the filter checks whether the session has expired. If so, the request is rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. `SASL_HANDSHAKE` / `SASL_AUTHENTICATE` are always permitted, allowing reauthentication. | + +#### Known limitations + +- The filter does not support SASL PLAIN (see [Rejected alternatives](#rejected-alternatives)). + +--- + +### Component 2: MechanismHandler internal extension point + +#### Summary + +The filter delegates the actual authentication exchange to mechanism-specific handlers, discovered via an internal extension point. This extension point provides internal extensibility for adding new mechanism support without modifying the filter itself. + +These are **not** user-facing plugins (no `@Plugin` annotation). The intention behind this decision is to encourage a small number of secure, high-quality implementations, one for each mechanism. Allowing pluggable implementations would make auditing for correctness and security significantly harder. + +#### API surfaces + +The extension point consists of three types, all in the `io.kroxylicious.filter.sasl.termination.mechanism` package within the `kroxylicious-sasl-termination` module. + +**`MechanismHandler`** -- handles the authentication exchange for a single connection. Instances are per-connection and not thread-safe. + +```java +public interface MechanismHandler { + + String mechanismName(); + + CompletionStage handleAuthenticate(byte[] authBytes); + + void dispose(); +} +``` + +**`MechanismHandlerFactory`** -- manages mechanism-specific resources and creates handler instances. Discovered via `ServiceLoader`. + +```java +public interface MechanismHandlerFactory extends AutoCloseable { + + String mechanismName(); + + void initialize(MechanismConfig config, FilterFactoryContext context, Clock clock) + throws PluginConfigurationException; + + MechanismHandler createHandler(); + + @Override + void close(); +} +``` + +Each factory: +1. Reports its IANA-registered mechanism name via `mechanismName()`. +2. Receives mechanism-specific configuration at `initialize()` time and creates whatever resources the mechanism requires (credential stores, JWKS callback handlers, etc.). +3. Creates per-connection `MechanismHandler` instances via `createHandler()`, injecting shared resources. +4. Releases resources on `close()`. + +**`AuthenticationResult`** -- the outcome of processing a single SASL authenticate request. + +```java +public record AuthenticationResult( + Outcome outcome, + byte[] responseBytes, + @Nullable String authorizationId, + @Nullable String errorMessage, + long sessionLifetimeMs) { + + public enum Outcome { CHALLENGE, SUCCESS, FAILURE } + + public static AuthenticationResult challenge(byte[] responseBytes); + public static AuthenticationResult success(byte[] responseBytes, String authorizationId); + public static AuthenticationResult success(byte[] responseBytes, String authorizationId, + long sessionLifetimeMs); + public static AuthenticationResult failure(byte[] responseBytes, String errorMessage); +} +``` + +**`MechanismConfig`** -- sealed interface for mechanism-specific configuration, using Jackson deduction-based polymorphism: + +```java +@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION) +@JsonSubTypes({ + @JsonSubTypes.Type(ScramMechanismConfig.class), + @JsonSubTypes.Type(OauthBearerMechanismConfig.class) +}) +public sealed interface MechanismConfig + permits ScramMechanismConfig, OauthBearerMechanismConfig { +} +``` + +#### ServiceLoader discovery + +Factories are registered in `META-INF/services/io.kroxylicious.filter.sasl.termination.mechanism.MechanismHandlerFactory`. At filter factory initialization time, the `SaslTermination` filter factory loads all registered factories, matches them to the mechanism names present in the user's configuration, and calls `initialize()` on each matched factory. + +#### Built-in mechanism handlers + +| Mechanism | Factory | Handler | Specification | +|-----------|---------|---------|---------------| +| `SCRAM-SHA-256` | `ScramSha256HandlerFactory` | `ScramHandler` | [RFC 5802][rfc5802] | +| `SCRAM-SHA-512` | `ScramSha512HandlerFactory` | `ScramHandler` | [RFC 5802][rfc5802] | +| `OAUTHBEARER` | `OauthBearerHandlerFactory` | `OauthBearerHandler` | [RFC 6750][rfc6750] / [RFC 7628][rfc7628] | + +#### Known limitations + +- Adding a new mechanism requires adding a new `MechanismHandlerFactory` implementation within the `kroxylicious-sasl-termination` module, a new `MechanismConfig` subtype, and updating the sealed permit list. This is intentional. + +--- + +### Component 3: SCRAM mechanism handler -The filter delegates the actual authentication exchange to mechanism-specific handlers, discovered via an internal extension point: +#### Summary -- `MechanismHandler` — handles the authentication exchange for a single connection. Implementations process `SaslAuthenticate` request bytes and return `AuthenticationResult` (CHALLENGE, SUCCESS, or FAILURE). Handlers are per-connection and not thread-safe. +The SCRAM mechanism handler (`ScramHandler`) implements multi-round SCRAM-SHA-256 and SCRAM-SHA-512 authentication by delegating to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework. Two factories -- `ScramSha256HandlerFactory` and `ScramSha512HandlerFactory` -- manage the credential store lifecycle and create per-connection handler instances. -- `MechanismHandlerFactory` — manages mechanism-specific resources and creates handler instances. Discovered via `ServiceLoader`. Each factory: - 1. Reports its IANA-registered mechanism name. - 2. Receives mechanism-specific configuration at `initialize()` time and creates whatever resources the mechanism requires (credential stores, JWKS callback handlers, etc.). - 3. Creates per-connection `MechanismHandler` instances, injecting shared resources. - 4. Releases resources on `close()`. +Key features: -These are **not** user-facing plugins (no `@Plugin` annotation). They provide internal extensibility for adding new mechanism support without modifying the filter itself. -The intention behind the decision **not** to make these user-facing plugins is to encourage a small number of secure, high-quality implementations, one for each mechanism. -Allowing pluggable implementations would make auditing for correctness and security significantly harder. +- **Multi-round SCRAM exchange.** SCRAM is a challenge-response protocol. The handler processes the client-first-message (round 1) and subsequent rounds, returning `CHALLENGE` until the exchange completes. +- **Delegation to Kafka's SaslServer.** The handler does not reimplement SCRAM. It creates a Kafka `SaslServer` with a `CallbackHandler` that supplies the looked-up credential, then processes all messages through it. This benefits from Kafka's battle-tested implementation. +- **Timing side-channel mitigation.** A fixed delay is applied to all authentication rounds to prevent attackers from distinguishing existing from non-existing users by measuring response times. -**Initial mechanism support:** +#### Authentication flow -| Mechanism | Handler | Notes | -|-----------|---------|-------| -| SCRAM-SHA-256 | `ScramHandler` via `ScramSha256HandlerFactory` | RFC 5802 | -| SCRAM-SHA-512 | `ScramHandler` via `ScramSha512HandlerFactory` | RFC 5802 | -| OAUTHBEARER | `OauthBearerHandler` via `OauthBearerHandlerFactory` | RFC 6750 / RFC 7628 | +1. **First round:** Extract the username from the SCRAM client-first-message, asynchronously look up the credential from the `ScramCredentialStore`, create a `SaslServer` with a `CallbackHandler` that supplies the credential, and process the first message. +2. **Subsequent rounds:** Process messages through the existing `SaslServer` synchronously. When `SaslServer.isComplete()` returns true, return `SUCCESS` with the authorization ID from `SaslServer.getAuthorizationID()`. -### OAUTHBEARER implementation +#### API surfaces -The OAUTHBEARER handler validates JWT bearer tokens at the proxy without forwarding them to the broker. +The SCRAM handler factories use: -The handler uses Kafka's `OAuthBearerValidatorCallbackHandler` for JWT validation, the same mechanism used by the existing OAUTHBEARER validation filter. The `OauthBearerHandlerFactory` manages the JWKS endpoint configuration and callback handler lifecycle: at `initialize()`-time it configures the callback handler with the JWKS endpoint, expected audience/issuer, and refresh settings; per-connection handlers receive the shared callback handler and use it to create a `SaslServer` via the JSSE/SASL framework. +- `MechanismHandlerFactory` / `MechanismHandler` (internal, Component 2) -- the internal extension point. +- `ScramCredentialStore` (public SPI, Component 5) -- for credential lookup. The factory resolves the credential store plugin at `initialize()` time using the Kroxylicious plugin system (`@PluginImplName` / `@PluginImplConfig`). -OAUTHBEARER is architecturally the simpler mechanism — it requires no credential store. The factory's only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). After successful authentication, the handler extracts the token's remaining lifetime from the `SaslServer`'s negotiated `CREDENTIAL.LIFETIME.MS` property for use in session lifetime computation (see [Reauthentication](#reauthentication-kip-368)). +#### Configuration + +SCRAM mechanisms are configured via `ScramMechanismConfig`: + +```java +public record ScramMechanismConfig( + @JsonProperty(required = true) + @PluginImplName(ScramCredentialStoreService.class) String credentialStore, + @JsonProperty(required = true) + @PluginImplConfig(implNameProperty = "credentialStore") Object credentialStoreConfig) + implements MechanismConfig { } +``` + +| Option | Type | Required | Default | Description | +|--------|------|----------|---------|-------------| +| `credentialStore` | `String` | Yes | -- | Plugin name of the `ScramCredentialStoreService` implementation (e.g. `KeystoreScramCredentialStoreService`). Resolved via the Kroxylicious plugin system. | +| `credentialStoreConfig` | `Object` | Yes | -- | Type-safe configuration for the credential store plugin. The actual type depends on the `credentialStore` plugin and is resolved via `@PluginImplConfig`. | + +#### Threats and mitigations + +| Threat | Mitigation | +|--------|------------| +| Username enumeration -- an attacker distinguishes existing from non-existing users by observing different error messages. | When a user is not found, the handler returns a generic `"Authentication failed"` error message identical to the message returned for incorrect credentials. | +| Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the handler applies a fixed delay to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. | +| SCRAM protocol correctness -- a bug in the SCRAM implementation could allow authentication bypass or credential leakage. | Delegated to Kafka's own `SaslServer`, which is widely deployed and well-tested. The handler is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. | + +#### Known limitations + +- **SCRAM channel binding not supported.** [RFC 5802 Section 6][rfc5802-s6] describes channel binding for SCRAM. Kafka does not use SCRAM channel binding, so this implementation follows Kafka's approach and does not implement it. + +--- + +### Component 4: OAUTHBEARER mechanism handler + +#### Summary + +The OAUTHBEARER mechanism handler (`OauthBearerHandler`) validates JWT bearer tokens at the proxy without forwarding them to the broker. The `OauthBearerHandlerFactory` manages the JWKS endpoint configuration and callback handler lifecycle. + +Key features: + +- **JWT validation via Kafka's `OAuthBearerValidatorCallbackHandler`.** The factory configures the callback handler at `initialize()` time with the JWKS endpoint, expected audience/issuer, and refresh settings. Per-connection handlers receive the shared callback handler and use it to create a `SaslServer` via the JSSE/SASL framework. +- **Token lifetime extraction for reauthentication.** After successful authentication, the handler extracts the token's remaining lifetime from the `SaslServer`'s negotiated `CREDENTIAL.LIFETIME.MS` property, returning it via `AuthenticationResult.sessionLifetimeMs` for use in session lifetime computation (see [Reauthentication](#reauthentication-kip-368)). +- **No credential store required.** OAUTHBEARER is architecturally simpler than SCRAM -- the factory's only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). **Key differences from the existing OAUTHBEARER validation filter:** -- The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, the termination handler validates tokens and _short-circuits_ — the broker never sees a SASL exchange. -- The handler factory owns its callback handler and JWKS configuration, receiving them at `initialize()`-time rather than requiring a credential store. +- The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, the termination handler validates tokens and _short-circuits_ -- the broker never sees a SASL exchange. +- The handler factory owns its callback handler and JWKS configuration, receiving them at `initialize()` time rather than requiring a credential store. -**Security requirements:** The `expectedAudience` and `expectedIssuer` fields are required. Without audience validation, a token issued for a different service would be accepted; without issuer validation, tokens from any issuer whose keys happen to be in the JWKS would be accepted. +#### API surfaces -**Known limitations:** -- **TLS configuration for the JWKS endpoint:** Kafka's `OAuthBearerValidatorCallbackHandler` uses an internal HTTP client with no TLS configuration surface. There is currently no way to configure custom trust stores or client certificates for HTTPS communication with the JWKS endpoint. The JVM's default trust store is used. This is a limitation inherited from Kafka's callback handler and shared with the existing OAUTHBEARER validation filter. -- **Rate limiting:** The handler does not implement rate limiting or brute-force protection for failed authentication attempts. The existing OAUTHBEARER validation filter has Caffeine-based rate limiting with exponential backoff that could serve as a reference for a future implementation. -- **Custom JWT validator:** The handler hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. +The OAUTHBEARER handler factory uses: -### SCRAM implementation +- `MechanismHandlerFactory` / `MechanismHandler` (internal, Component 2) -- the internal extension point. -SCRAM is more complex than OAUTHBEARER because it is a multi-round challenge-response protocol that requires stored credentials. +It does not use the `ScramCredentialStore` SPI. Token validation is performed entirely by Kafka's `OAuthBearerValidatorCallbackHandler`. -The SCRAM handler delegates to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework: +#### Configuration -1. **First round:** Extract the username from the SCRAM client-first-message, asynchronously look up the credential from the store, create a `SaslServer` with a `CallbackHandler` that supplies the credential, and process the first message. +OAUTHBEARER is configured via `OauthBearerMechanismConfig`: -2. **Subsequent rounds:** Process messages through the existing `SaslServer` synchronously. When `SaslServer.isComplete()` returns true, return SUCCESS with the authorization ID from `SaslServer.getAuthorizationID()`. +| Option | Type | Required | Default | Description | +|--------|------|----------|---------|-------------| +| `jwksEndpointUrl` | `URI` | Yes | -- | URL of the JWKS endpoint for fetching token signing keys. | +| `expectedAudience` | `String` | Yes | -- | Expected `aud` claim value. Comma-separated for multiple audiences. Tokens without a matching audience are rejected. | +| `expectedIssuer` | `String` | Yes | -- | Expected `iss` claim value. Tokens from a different issuer are rejected. | +| `scopeClaimName` | `String` | No | `"scope"` | JWT claim name containing the scope. | +| `subClaimName` | `String` | No | `"sub"` | JWT claim name containing the subject. | +| `jwksEndpointRefreshMs` | `Long` | No | Kafka default | Interval in milliseconds between JWKS endpoint refreshes. | +| `jwksEndpointRetryBackoffMs` | `Long` | No | Kafka default | Initial retry backoff in milliseconds when the JWKS endpoint is unreachable. | +| `jwksEndpointRetryBackoffMaxMs` | `Long` | No | Kafka default | Maximum retry backoff in milliseconds. | -This approach avoids reimplementing the SCRAM protocol and benefits from Kafka's battle-tested implementation. +**Security note:** `expectedAudience` and `expectedIssuer` are both required. Without audience validation, a token issued for a different service would be accepted; without issuer validation, tokens from any issuer whose keys happen to be in the JWKS would be accepted. -#### SCRAM Credential store SPI +#### Threats and mitigations -SCRAM mechanisms need a way to look up stored credentials. The credential store SPI provides async credential lookup, decoupled from any particular storage backend. +| Threat | Mitigation | +|--------|------------| +| Token from wrong audience or issuer -- a JWT issued for a different service or identity provider is presented to the proxy. | Both `expectedAudience` and `expectedIssuer` are required fields. The handler rejects tokens that do not match. | +| JWKS endpoint compromise -- an attacker controls the JWKS endpoint and supplies signing keys for forged tokens. | Mitigated operationally: the JWKS endpoint URL is set by the proxy administrator, not by clients. TLS protects the endpoint in transit (using the JVM's default trust store). | -**Core types:** +#### Known limitations -- `ScramCredentialStore` — the lookup interface, returning `CompletionStage` for a given username. Returns `null` (via completed stage) when the user is not found. Exceptional completions indicate infrastructure failures. +- **No TLS configuration for the JWKS endpoint.** Kafka's `OAuthBearerValidatorCallbackHandler` uses an internal HTTP client with no TLS configuration surface. There is no way to configure custom trust stores or client certificates for HTTPS communication with the JWKS endpoint. The JVM's default trust store is used. This limitation is inherited from Kafka's callback handler and shared with the existing OAUTHBEARER validation filter. +- **No rate limiting.** The handler does not implement rate limiting or brute-force protection for failed authentication attempts. The existing OAUTHBEARER validation filter has Caffeine-based rate limiting with exponential backoff that could serve as a reference for a future implementation. +- **Hardcoded `BrokerJwtValidator`.** The handler hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. -- `ScramCredentialStoreService` — the lifecycle interface for credential store providers. Follows the initialize/build/close pattern used by `KmsService`: - 1. `initialize(C config)` — validate and store configuration. - 2. `buildCredentialStore()` — create an operational store instance. - 3. `close()` — release resources. +--- -- `ScramCredential` — an immutable sealed record holding the username, salt, iteration count, server key, stored key, and hash algorithm. Byte array fields use defensive copies in the constructor and accessors to prevent mutation. The `toString()` method redacts sensitive fields. +### Component 5: ScramCredentialStore SPI (public plugin API) -- Exception hierarchy: `CredentialLookupException` with subtypes `CredentialServiceUnavailableException` and `CredentialServiceTimeoutException`. +#### Summary -**Design note:** The SPI is intentionally SCRAM-specific. OAUTHBEARER uses token validation against a JWKS endpoint, which has a fundamentally different shape from stored credential lookup. Rather than creating a leaky abstraction that covers both, each mechanism family uses its own resource management approach (see [Rejected alternatives](#rejected-alternatives)). +The `ScramCredentialStore` SPI, defined in the `kroxylicious-sasl-credential-store` module, is the user-facing plugin API for SCRAM credential store providers. It provides asynchronous credential lookup decoupled from any particular storage backend. -#### `KeyStore`-based credential store provider +The SPI is intentionally SCRAM-specific. OAUTHBEARER uses token validation against a JWKS endpoint, which has a fundamentally different shape from stored credential lookup. Rather than creating a leaky abstraction that covers both, each mechanism family uses its own resource management approach (see [Rejected alternatives](#rejected-alternatives)). -The first-party provider stores SCRAM credentials in a Java `KeyStore` file, following the project's established pattern of using `KeyStores` to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry keyed by username. +**No Kafka type dependencies.** The SPI types (`ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, exception hierarchy) do not reference any Kafka types. Implementors of the credential store SPI are not transitively exposed to Kafka internal APIs. -**Characteristics:** +#### API surfaces -- Loads the entire KeyStore into memory at construction time for sub-millisecond lookups. -- Does not support hot reloading — credential changes require a proxy restart or virtual cluster reconfiguration. -- Supports PKCS12 and JKS store types. -- Uses the Kroxylicious `PasswordProvider` abstraction for KeyStore and key passwords, supporting both file-based (production) and inline (development) password configuration. +All types are in the `io.kroxylicious.sasl.credentialstore` package. -**CLI credential management tool** (`KeystoreCredentialTool`): +**`ScramCredentialStore`** -- the lookup interface: -The credentials stored in the KeyStore are serialized JSON, which makes for less than ideal UX: The user needs ensure the JSON has the required format. -Moreover, the values of that JSON are not all obvious things like the username. Some of the fields are computed from cryptographic operation on the password which need to -be done correctly for the authentication to work, and where incorrect construction can undermine security. +```java +public interface ScramCredentialStore { -To provide a better UX and to reduce the possibility of user error compromising security a PicoCLI-based command-line tool will be provided for managing credentials in KeyStore files. Supports: `create`, `add-user`, `remove-user`, `update-password`, `list-users`. + CompletionStage lookupCredential(String username); +} +``` -Security measures: -- Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure. Command-line password arguments are supported, but gated behind an `--unlock-insecure-options` flag that displays security warnings. -- A 12-character minimum password length is enforced, following [NIST SP 800-63B][nist-sp800-63b] guidance. -- SCRAM credentials are generated with 10,000 iterations (above the RFC-5802 minimum of 4,096) and 20 bytes of random salt. +Returns a `CompletionStage` that completes with: +- A `ScramCredential` if the user exists. +- `null` if the user does not exist. +- Exceptional completion with `CredentialLookupException` (or subtype) on infrastructure failure. -### Configuration model +**`ScramCredentialStoreService`** -- the lifecycle interface for credential store providers. Follows the same initialize/build/close pattern used by `KmsService`: -```yaml -filters: - - type: SaslTermination - config: - maxTimeBeforeReauth: 1h - mechanisms: - SCRAM-SHA-256: - credentialStore: KeystoreScramCredentialStoreService - credentialStoreConfig: - file: /path/to/credentials.p12 - storePassword: - file: /etc/kroxylicious/keystore-password.txt - storeType: PKCS12 - OAUTHBEARER: - jwksEndpointUrl: https://idp.example.com/.well-known/jwks.json - expectedAudience: kafka - expectedIssuer: https://idp.example.com +```java +public interface ScramCredentialStoreService extends AutoCloseable { + + void initialize(C config); + + ScramCredentialStore buildCredentialStore() throws IllegalStateException; + + @Override + default void close() { } +} ``` -The `mechanisms` map is keyed by IANA-registered mechanism name. The config shape for each entry depends on the mechanism: SCRAM mechanisms use `credentialStore`/`credentialStoreConfig`, while OAUTHBEARER uses JWKS endpoint configuration directly. +Lifecycle: +1. `initialize(C config)` -- validate and store configuration. Called exactly once. +2. `buildCredentialStore()` -- create an operational store instance. May be called multiple times. +3. `close()` -- release resources. Must be idempotent. Must tolerate being called on an uninitialized or partially initialized service. + +**`ScramCredential`** -- immutable record holding the SCRAM credential data: + +```java +public record ScramCredential( + String username, + byte[] salt, + int iterations, + byte[] serverKey, + byte[] storedKey, + String hashAlgorithm) { + + public static final int MINIMUM_ITERATIONS = 4096; + // Supported: "SHA-256", "SHA-512" +} +``` + +Security properties: +- `byte[]` fields (`salt`, `serverKey`, `storedKey`) use defensive copies in both the constructor and accessors. +- `toString()` redacts sensitive fields (salt, serverKey, storedKey). +- `iterations` must be at least 4096 (RFC 5802 minimum). +- `hashAlgorithm` must be `"SHA-256"` or `"SHA-512"`. + +**Exception hierarchy:** + +```java +public class CredentialLookupException extends Exception { ... } + +public class CredentialServiceUnavailableException extends CredentialLookupException { ... } + +public class CredentialServiceTimeoutException extends CredentialLookupException { ... } +``` + +- `CredentialLookupException` -- base exception for credential lookup failures (service-level issues, not user-not-found). +- `CredentialServiceUnavailableException` -- backing service is unavailable (database down, LDAP unreachable, etc.). +- `CredentialServiceTimeoutException` -- lookup operation timed out. + +#### Known limitations + +- The SPI covers only SCRAM credential lookup. There is no generic credential store abstraction spanning mechanisms. This is a deliberate design decision (see [Rejected alternatives](#rejected-alternatives)). + +--- + +### Component 6: KeyStore credential store provider + +#### Summary + +The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry keyed by username. + +Key features: + +- **PKCS12 and JKS support.** Both KeyStore types are supported. +- **In-memory loading.** The entire KeyStore is loaded into memory at construction time for sub-millisecond lookups. +- **`PasswordProvider` for secrets.** Uses the Kroxylicious `PasswordProvider` abstraction for KeyStore and key passwords, supporting both file-based (production) and inline (development) password configuration. +- **File permission enforcement.** On POSIX systems, the credential store refuses to load a KeyStore file that has group or world read/write permissions. This prevents accidental exposure through overly permissive file modes. + +#### API surfaces + +- Implements `ScramCredentialStoreService` (from Component 5). +- Annotated with `@Plugin` for discovery by the Kroxylicious plugin system. + +#### Configuration + +The provider is configured via `KeystoreScramCredentialStoreConfig`: + +| Option | Type | Required | Default | Description | +|--------|------|----------|---------|-------------| +| `file` | `String` | Yes | -- | Path to the Java KeyStore file. | +| `storePassword` | `PasswordProvider` | Yes | -- | Password provider for the KeyStore. In production, use file-based password (`file: /path/to/password.txt`). | +| `keyPassword` | `PasswordProvider` | No | value of `storePassword` | Password provider for individual keys within the KeyStore. Defaults to `storePassword` if not specified. | +| `storeType` | `String` | No | `KeyStore.getDefaultType()` | KeyStore type (e.g. `PKCS12`, `JKS`). Defaults to the JVM platform default. | + +#### CLI tool: `KeystoreCredentialTool` + +The credentials stored in the KeyStore are serialized JSON, which makes for less than ideal UX: the user needs to ensure the JSON has the required format. Moreover, some fields are computed from cryptographic operations on the password which must be done correctly for authentication to work, and where incorrect construction can undermine security. + +To provide a better UX and to reduce the possibility of user error compromising security, a PicoCLI-based command-line tool is provided for managing credentials in KeyStore files. + +**Commands:** + +| Command | Description | +|---------|-------------| +| `create` | Create a new KeyStore file. | +| `add-user` | Add a SCRAM credential for a user. | +| `remove-user` | Remove a user's credential. | +| `update-password` | Update a user's password (recomputes SCRAM credential). | +| `list-users` | List all usernames in the KeyStore. | + +**Security measures:** +- Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure (they appear in shell history and process listings). Command-line password arguments are supported but gated behind an `--unlock-insecure-options` flag that displays security warnings. +- A 12-character minimum password length is enforced, following [NIST SP 800-63B][nist-sp800-63b] guidance. +- SCRAM credentials are generated with 10,000 iterations (above the RFC 5802 minimum of 4,096) and 20 bytes of random salt. + +#### Threats and mitigations + +| Threat | Mitigation | +|--------|------------| +| KeyStore file exposure -- an attacker gains read access to the KeyStore file on disk. | POSIX file permission check: the provider refuses to load a KeyStore with group or world read/write permissions. The KeyStore itself is password-encrypted. | +| Credential material in memory -- sensitive key material (serverKey, storedKey, salt) is accessible in the JVM heap. | `ScramCredential` uses defensive copies for all `byte[]` fields in both the constructor and accessors, preventing callers from mutating stored data. `toString()` redacts sensitive fields. | + +#### Known limitations + +- **No hot reloading.** Credential changes require a proxy restart or virtual cluster reconfiguration. The KeyStore is loaded once at construction time. -The optional `maxTimeBeforeReauth` sets the maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. +--- -### Module architecture +### Component 7: Module architecture The implementation is organized into three modules, following the same pattern as the existing KMS modules (`kroxylicious-kms`, `kroxylicious-kms-provider-*`): -| Module | Purpose | -|--------|---------| -| `kroxylicious-filters/kroxylicious-sasl-termination` | The termination filter, state machine, mechanism handler extensibility, and all built-in mechanism implementations | -| `kroxylicious-sasl-credential-store` | Public API: defines the credential store SPI used by SCRAM mechanism handlers | -| `kroxylicious-sasl-credential-store-providers/kroxylicious-sasl-credential-store-provider-keystore` | First-party SCRAM credential provider: Java KeyStore-backed implementation with CLI management tool | +| Module | Contents | Components | +|--------|----------|------------| +| `kroxylicious-filters/kroxylicious-sasl-termination` | Filter, state machine, `MechanismHandler` / `MechanismHandlerFactory` internal SPI, `MechanismConfig` sealed hierarchy, and all built-in mechanism handler implementations (SCRAM, OAUTHBEARER). | 1, 2, 3, 4 | +| `kroxylicious-sasl-credential-store` | Public API: `ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, exception hierarchy. No implementation, no Kafka dependencies. | 5 | +| `kroxylicious-sasl-credential-store-providers/kroxylicious-sasl-credential-store-provider-keystore` | First-party SCRAM credential provider: Java KeyStore-backed `ScramCredentialStoreService` implementation with `KeystoreCredentialTool` CLI. | 6 | ## Security model From 14ca471ca13b69cb6f231b82d865793493082560 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 23:16:44 +0000 Subject: [PATCH 10/52] docs(proposal): clarify heap exposure as accepted risk, not mitigated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Defensive byte[] copies and toString redaction are correctness measures (preventing mutation bugs and log leakage), not security mitigations. Credential material in the JVM heap is an accepted risk — there is no practical mitigation within a JVM. Documents operational controls (JMX auth, core dump permissions, container policies) as the defence. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index f0d1e040..b4fc49c3 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -170,7 +170,7 @@ filters: The filter delegates the actual authentication exchange to mechanism-specific handlers, discovered via an internal extension point. This extension point provides internal extensibility for adding new mechanism support without modifying the filter itself. -These are **not** user-facing plugins (no `@Plugin` annotation). The intention behind this decision is to encourage a small number of secure, high-quality implementations, one for each mechanism. Allowing pluggable implementations would make auditing for correctness and security significantly harder. +These are **not** intended to be configurable by end uses (no `@Plugin` annotation). The intention behind this decision is to encourage a small number of secure, high-quality implementations, one for each mechanism. Allowing pluggable implementations would make auditing for correctness and security significantly harder. #### API surfaces @@ -267,12 +267,12 @@ Factories are registered in `META-INF/services/io.kroxylicious.filter.sasl.termi #### Summary -The SCRAM mechanism handler (`ScramHandler`) implements multi-round SCRAM-SHA-256 and SCRAM-SHA-512 authentication by delegating to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework. Two factories -- `ScramSha256HandlerFactory` and `ScramSha512HandlerFactory` -- manage the credential store lifecycle and create per-connection handler instances. +The SCRAM mechanism handler (`ScramHandler`) implements multi-round `SCRAM-SHA-256` and `SCRAM-SHA-512` authentication by delegating to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework. Two factories -- `ScramSha256HandlerFactory` and `ScramSha512HandlerFactory` -- manage the credential store lifecycle and create per-connection handler instances. Key features: - **Multi-round SCRAM exchange.** SCRAM is a challenge-response protocol. The handler processes the client-first-message (round 1) and subsequent rounds, returning `CHALLENGE` until the exchange completes. -- **Delegation to Kafka's SaslServer.** The handler does not reimplement SCRAM. It creates a Kafka `SaslServer` with a `CallbackHandler` that supplies the looked-up credential, then processes all messages through it. This benefits from Kafka's battle-tested implementation. +- **Delegation to Kafka's `SaslServer`.** The handler does not reimplement SCRAM. It creates a Kafka `SaslServer` with a `CallbackHandler` that supplies the looked-up credential, then processes all messages through it. This benefits from Kafka's battle-tested implementation. - **Timing side-channel mitigation.** A fixed delay is applied to all authentication rounds to prevent attackers from distinguishing existing from non-existing users by measuring response times. #### Authentication flow @@ -519,7 +519,10 @@ To provide a better UX and to reduce the possibility of user error compromising | Threat | Mitigation | |--------|------------| | KeyStore file exposure -- an attacker gains read access to the KeyStore file on disk. | POSIX file permission check: the provider refuses to load a KeyStore with group or world read/write permissions. The KeyStore itself is password-encrypted. | -| Credential material in memory -- sensitive key material (serverKey, storedKey, salt) is accessible in the JVM heap. | `ScramCredential` uses defensive copies for all `byte[]` fields in both the constructor and accessors, preventing callers from mutating stored data. `toString()` redacts sensitive fields. | + +**Accepted risk: credential material in JVM heap.** SCRAM credential data (serverKey, storedKey, salt) is held in memory for the lifetime of the proxy. An attacker who can obtain a heap dump (e.g. via JMX, `/proc//mem`, or a core dump) can extract this material. There is no practical mitigation within a JVM — `byte[]` contents cannot be reliably zeroed because the GC may copy them, and off-heap storage would add complexity without eliminating the risk. Operators should protect heap dump access through operational controls (JMX authentication, file permissions on core dumps, container security policies). + +Note: `ScramCredential` uses defensive copies for `byte[]` fields and redacts `toString()`, but these are correctness measures (preventing accidental mutation and log leakage), not security mitigations against heap inspection. #### Known limitations @@ -544,7 +547,7 @@ The implementation is organized into three modules, following the same pattern a - **KeyStore encryption:** Credentials are stored in Java KeyStore files, encrypted with the KeyStore password. File-system permissions and KeyStore passwords are the primary access controls. - **PasswordProvider abstraction:** Production deployments should use file-based passwords rather than inline passwords in configuration. The `PasswordProvider` interface supports both. - **File permission enforcement:** On POSIX systems, the credential store refuses to load a KeyStore file that has group or world read/write permissions. This prevents accidental exposure of credential material through overly permissive file modes. -- **In-memory handling:** `ScramCredential` uses defensive copies for all `byte[]` fields (salt, serverKey, storedKey) in both the constructor and accessors, preventing callers from mutating stored credential data. `toString()` redacts sensitive fields. +- **In-memory handling:** `ScramCredential` uses defensive copies for `byte[]` fields (correctness measure against accidental mutation) and `toString()` redacts sensitive fields (prevents log leakage). Credential material in the JVM heap is an accepted risk — see Component 6 threat discussion. ### SCRAM protocol correctness From d3fe03d4441b9a0ac910d30f489d5a75baac6043 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 23:20:01 +0000 Subject: [PATCH 11/52] docs(proposal): clarify PBKDF2 iteration count rationale 10,000 iterations is above Kafka's default (4,096) and the RFC 5802 minimum, but below OWASP's current recommendation of 600,000. The difference is justified: OWASP targets password storage where derivation happens once, while SCRAM clients perform derivation on every authentication. Documents the tradeoff explicitly. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index b4fc49c3..191b09e7 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -512,7 +512,7 @@ To provide a better UX and to reduce the possibility of user error compromising **Security measures:** - Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure (they appear in shell history and process listings). Command-line password arguments are supported but gated behind an `--unlock-insecure-options` flag that displays security warnings. - A 12-character minimum password length is enforced, following [NIST SP 800-63B][nist-sp800-63b] guidance. -- SCRAM credentials are generated with 10,000 iterations (above the RFC 5802 minimum of 4,096) and 20 bytes of random salt. +- SCRAM credentials are generated with 10,000 PBKDF2 iterations and 20 bytes of random salt. The [RFC 5802][rfc5802] minimum is 4,096 (which is also the Kafka broker default). The [OWASP Password Storage Cheat Sheet][owasp-password-storage] currently recommends 600,000 iterations for PBKDF2-HMAC-SHA256, but that guidance targets password storage hashing where derivation happens once at write time. In SCRAM, the client performs the derivation on every authentication, so the iteration count directly affects authentication latency. 10,000 provides a reasonable balance between brute-force resistance and authentication performance for Kafka's typically long-lived connections. #### Threats and mitigations @@ -520,7 +520,7 @@ To provide a better UX and to reduce the possibility of user error compromising |--------|------------| | KeyStore file exposure -- an attacker gains read access to the KeyStore file on disk. | POSIX file permission check: the provider refuses to load a KeyStore with group or world read/write permissions. The KeyStore itself is password-encrypted. | -**Accepted risk: credential material in JVM heap.** SCRAM credential data (serverKey, storedKey, salt) is held in memory for the lifetime of the proxy. An attacker who can obtain a heap dump (e.g. via JMX, `/proc//mem`, or a core dump) can extract this material. There is no practical mitigation within a JVM — `byte[]` contents cannot be reliably zeroed because the GC may copy them, and off-heap storage would add complexity without eliminating the risk. Operators should protect heap dump access through operational controls (JMX authentication, file permissions on core dumps, container security policies). +**Accepted risk: credential material in JVM heap.** SCRAM credential data (serverKey, storedKey, salt) is held in memory for the lifetime of the proxy. An attacker who can obtain a heap dump (e.g. via JMX, `/proc//mem`, or a core dump) can extract this material. There is no practical mitigation within a JVM. Operators should protect heap dump access through operational controls (JMX authentication, file permissions on core dumps, container security policies). Note: `ScramCredential` uses defensive copies for `byte[]` fields and redacts `toString()`, but these are correctness measures (preventing accidental mutation and log leakage), not security mitigations against heap inspection. @@ -680,5 +680,6 @@ Supporting SASL PLAIN was deferred because: [kip255]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75968876 [kip368]: https://cwiki.apache.org/confluence/spaces/KAFKA/pages/89068981/KIP-368+Allow+SASL+Connections+to+Periodically+Re-Authenticate [nist-sp800-63b]: https://pages.nist.gov/800-63-4/sp800-63b.html +[owasp-password-storage]: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html [sasl-inspection]: https://kroxylicious.io/kroxylicious/#assembly-sasl-inspection [oauthbearer-validation]: https://kroxylicious.io/kroxylicious/#assembly-configuring-oauth-bearer-validation-filter From a22b862c5a3d844043e5e69c99121121f2c93877 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 23:32:11 +0000 Subject: [PATCH 12/52] docs(proposal): document CLI tool interface in man-page style Each subcommand now shows its synopsis, options table with required/ default/description columns, and exit codes. Replaces the previous summary table with full per-command documentation. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 81 ++++++++++++++++++++++++++++--- 1 file changed, 74 insertions(+), 7 deletions(-) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index 191b09e7..fd231b4e 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -499,15 +499,82 @@ The credentials stored in the KeyStore are serialized JSON, which makes for less To provide a better UX and to reduce the possibility of user error compromising security, a PicoCLI-based command-line tool is provided for managing credentials in KeyStore files. +**Global options:** + +``` +keystore-credential-tool [--unlock-insecure-options] [options] +``` + +| Option | Description | +|--------|-------------| +| `--unlock-insecure-options` | Enable command-line password options (`-p`, `-w`). Without this flag, passwords must be entered via interactive console prompts. Displays security warnings when used. | + **Commands:** -| Command | Description | -|---------|-------------| -| `create` | Create a new KeyStore file. | -| `add-user` | Add a SCRAM credential for a user. | -| `remove-user` | Remove a user's credential. | -| `update-password` | Update a user's password (recomputes SCRAM credential). | -| `list-users` | List all usernames in the KeyStore. | +``` +keystore-credential-tool create -k [-p ] [-t ] +``` + +Create a new, empty KeyStore file. + +| Option | Required | Default | Description | +|--------|----------|---------|-------------| +| `-k`, `--keystore` | Yes | — | Path to the KeyStore file to create. | +| `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | +| `-t`, `--type` | No | `PKCS12` | KeyStore type (`PKCS12`, `JKS`). | + +``` +keystore-credential-tool add-user -k -u [-p ] [-w ] [-m ] +``` + +Add a SCRAM credential for a user. If the user already exists, their credential is replaced. + +| Option | Required | Default | Description | +|--------|----------|---------|-------------| +| `-k`, `--keystore` | Yes | — | Path to the KeyStore file. | +| `-u`, `--username` | Yes | — | Username to add. | +| `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | +| `-w`, `--user-password` | No | interactive prompt | User's password. Requires `--unlock-insecure-options`. | +| `-m`, `--mechanism` | No | `SCRAM_SHA_256` | SCRAM mechanism (`SCRAM_SHA_256`, `SCRAM_SHA_512`). | + +``` +keystore-credential-tool remove-user -k -u [-p ] +``` + +Remove a user's credential from the KeyStore. + +| Option | Required | Default | Description | +|--------|----------|---------|-------------| +| `-k`, `--keystore` | Yes | — | Path to the KeyStore file. | +| `-u`, `--username` | Yes | — | Username to remove. | +| `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | + +``` +keystore-credential-tool update-password -k -u [-p ] [-w ] [-m ] +``` + +Update a user's password. Recomputes the SCRAM credential with a new salt. + +| Option | Required | Default | Description | +|--------|----------|---------|-------------| +| `-k`, `--keystore` | Yes | — | Path to the KeyStore file. | +| `-u`, `--username` | Yes | — | Username to update. | +| `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | +| `-w`, `--new-password` | No | interactive prompt | New password for the user. Requires `--unlock-insecure-options`. | +| `-m`, `--mechanism` | No | `SCRAM_SHA_256` | SCRAM mechanism (`SCRAM_SHA_256`, `SCRAM_SHA_512`). | + +``` +keystore-credential-tool list-users -k [-p ] +``` + +List all usernames in the KeyStore. + +| Option | Required | Default | Description | +|--------|----------|---------|-------------| +| `-k`, `--keystore` | Yes | — | Path to the KeyStore file. | +| `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | + +**Exit codes:** `0` = success, `1` = operation error, `2` = password/security error. **Security measures:** - Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure (they appear in shell history and process listings). Command-line password arguments are supported but gated behind an `--unlock-insecure-options` flag that displays security warnings. From c5f300fc8c63527690c7856bcb606944674b4846 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 20 Jul 2026 23:36:06 +0000 Subject: [PATCH 13/52] docs(proposal): document CLI tool file permission security measures New keystores are created with owner-only permissions (rw-------). Existing keystores are checked for insecure permissions before modification. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/000-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/000-sasl-termination.md b/proposals/000-sasl-termination.md index fd231b4e..d12eb647 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/000-sasl-termination.md @@ -580,6 +580,7 @@ List all usernames in the KeyStore. - Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure (they appear in shell history and process listings). Command-line password arguments are supported but gated behind an `--unlock-insecure-options` flag that displays security warnings. - A 12-character minimum password length is enforced, following [NIST SP 800-63B][nist-sp800-63b] guidance. - SCRAM credentials are generated with 10,000 PBKDF2 iterations and 20 bytes of random salt. The [RFC 5802][rfc5802] minimum is 4,096 (which is also the Kafka broker default). The [OWASP Password Storage Cheat Sheet][owasp-password-storage] currently recommends 600,000 iterations for PBKDF2-HMAC-SHA256, but that guidance targets password storage hashing where derivation happens once at write time. In SCRAM, the client performs the derivation on every authentication, so the iteration count directly affects authentication latency. 10,000 provides a reasonable balance between brute-force resistance and authentication performance for Kafka's typically long-lived connections. +- On POSIX systems, newly created KeyStore files are set to owner-only permissions (`rw-------`). When loading an existing KeyStore for modification (`add-user`, `remove-user`, `update-password`, `list-users`), the tool checks that the file does not have group or world read/write permissions and refuses to proceed if it does. #### Threats and mitigations From d34999e827b292ae431f960c8c12f86253801b22 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 21 Jul 2026 13:47:07 +1200 Subject: [PATCH 14/52] Rename proposal to use PR number 124 Signed-off-by: Tom Bentley --- proposals/{000-sasl-termination.md => 124-sasl-termination.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{000-sasl-termination.md => 124-sasl-termination.md} (99%) diff --git a/proposals/000-sasl-termination.md b/proposals/124-sasl-termination.md similarity index 99% rename from proposals/000-sasl-termination.md rename to proposals/124-sasl-termination.md index d12eb647..9657b1be 100644 --- a/proposals/000-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -1,4 +1,4 @@ -# 000 - SASL Termination +# 124 - SASL Termination SASL termination allows the Kroxylicious proxy to authenticate Kafka clients directly, without forwarding SASL exchanges to the upstream Kafka broker. This enables credential isolation, authentication protocol translation, and centralized credential management. From 25ae2803370603309b851282abbb36af0bd9a385 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 21 Jul 2026 02:13:28 +0000 Subject: [PATCH 15/52] docs(proposal): explain why mechanisms belong in a single filter The Kafka SASL protocol requires the server to advertise all supported mechanisms in SaslHandshakeResponse. A filter-per-mechanism model would not work because no single filter would have the complete set. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 9657b1be..e5ca149f 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -61,7 +61,7 @@ The proposal is organized per-component. Each component section covers its summa #### Summary -The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, authenticating clients at the proxy and short-circuiting the responses without forwarding them to the broker. +The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, authenticating clients at the proxy and short-circuiting the responses without forwarding them to the broker. Multiple mechanisms are configured within a single filter instance because the Kafka SASL protocol requires it: the client sends a `SaslHandshakeRequest` naming its chosen mechanism, and the server responds with the set of supported mechanisms. A filter-per-mechanism model would not work because no single filter would have the complete set of supported mechanisms to advertise in the `SaslHandshakeResponse`. Key features: From 6d764598bdaeed1998c81e1f3400e36595e6e071 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 21 Jul 2026 02:15:48 +0000 Subject: [PATCH 16/52] docs(proposal): replace ASCII state diagram with transition table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ASCII art diagram was ambiguous — edge labels looked like nodes, and RequiringAuthenticate appeared twice. A transition table is unambiguous: each row is one edge with from-state, event, and to-state. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index e5ca149f..ceda631f 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -74,17 +74,16 @@ Key features: The filter maintains per-connection state using a sealed interface `State` with four concrete states: -``` -RequiringHandshake ──> RequiringAuthenticate <────╮ - | | - |─> (multi-round) ──╯ - | - |──> Authenticated ──> (reauth) ──> RequiringAuthenticate - | | - | └──> (expired + non-SASL request) ──> reject & close - | - └──> Failed (terminal) -``` +| From state | Event | To state | +|------------|-------|----------| +| **RequiringHandshake** | `SASL_HANDSHAKE` with supported mechanism | **RequiringAuthenticate** | +| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `CHALLENGE` | **RequiringAuthenticate** (loop) | +| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `SUCCESS` | **Authenticated** | +| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `FAILURE` | **Failed** | +| **Authenticated** | `SASL_HANDSHAKE` (reauthentication) | **RequiringAuthenticate** | +| **Authenticated** | non-SASL request, session not expired | forward to broker | +| **Authenticated** | non-SASL request, session expired | reject and close | +| **Failed** | *(terminal — connection closed)* | — | - **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests, which negotiate the mechanism and transition to `RequiringAuthenticate`. - **RequiringAuthenticate:** Accepts `SASL_AUTHENTICATE` requests. Loops back to itself for multi-round mechanisms (e.g. SCRAM). Carries a reference to the `MechanismHandler` for the negotiated mechanism. From e2efa79eb602fbfc565db8d9b7eaf5b3cf1bd222 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 21 Jul 2026 02:18:35 +0000 Subject: [PATCH 17/52] docs(proposal): rename Event column to Triggering event for clarity Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index ceda631f..5fbd72f1 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -74,8 +74,8 @@ Key features: The filter maintains per-connection state using a sealed interface `State` with four concrete states: -| From state | Event | To state | -|------------|-------|----------| +| From state | Triggering event | To state | +|------------|------------------|----------| | **RequiringHandshake** | `SASL_HANDSHAKE` with supported mechanism | **RequiringAuthenticate** | | **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `CHALLENGE` | **RequiringAuthenticate** (loop) | | **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `SUCCESS` | **Authenticated** | From 078e71b7dbf0bb10195c9f1abf05309fae9d9907 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 21 Jul 2026 02:21:54 +0000 Subject: [PATCH 18/52] docs(proposal): document per-mechanism reauthentication behaviour MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SCRAM credentials don't expire, so maxTimeBeforeReauth is the sole session lifetime source. OAUTHBEARER tokens have inherent expiry, so the effective lifetime is min(config, tokenExpiry). All mechanisms support reauthentication — it's protocol-level, not mechanism-specific. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 5fbd72f1..40dac6e7 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -100,6 +100,10 @@ The filter supports [KIP-368][kip368] reauthentication. If either value is zero (no opinion / no expiry), the other is used. If both are zero, no reauthentication is required. +Reauthentication is a protocol-level feature, not mechanism-specific — all mechanisms support it. The difference is the session lifetime source: +- **SCRAM:** Credentials do not expire, so the handler reports no lifetime. `maxTimeBeforeReauth` is the sole source of session lifetime. Without it configured, SCRAM sessions never require reauthentication. +- **OAUTHBEARER:** Tokens have an inherent expiry. The handler reports the token's remaining lifetime, and the effective session lifetime is `min(maxTimeBeforeReauth, tokenExpiry)`. Even without `maxTimeBeforeReauth`, sessions expire when the token does. + **Client behaviour:** Standard Kafka clients (4.0+) handle reauthentication transparently via the `Selector`. When the session nears expiry, the client sends a new `SASL_HANDSHAKE` + `SASL_AUTHENTICATE` sequence over the existing connection. This is invisible to application code. **Server-side enforcement:** If the session has expired and a non-SASL request arrives, the filter rejects it with `SASL_AUTHENTICATION_FAILED` and closes the connection. `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests are always accepted regardless of session expiry, to allow reauthentication. From 76a8cd21171746b72ed852ad6ba8ddce09f89efd Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 21 Jul 2026 02:24:57 +0000 Subject: [PATCH 19/52] docs(proposal): add GSSAPI (Kerberos) to rejected alternatives GSSAPI requires service principal keytabs and Kerberos infrastructure participation, which is a fundamentally different operational model. Terminating Kerberos at the proxy raises complex delegation and trust questions that are out of scope for this proposal. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 40dac6e7..e6e866e8 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -163,7 +163,7 @@ filters: #### Known limitations -- The filter does not support SASL PLAIN (see [Rejected alternatives](#rejected-alternatives)). +- The filter does not support SASL PLAIN or GSSAPI (Kerberos). See [Rejected alternatives](#rejected-alternatives). --- @@ -726,6 +726,14 @@ Supporting SASL PLAIN was deferred because: - SCRAM provides mutual authentication and never transmits the password (though should also be used with TLS to avoid MitM attacks). - If PLAIN support is needed in the future, it could be added as a new `MechanismHandler` implementation. +### GSSAPI (Kerberos) mechanism support + +Supporting SASL GSSAPI was deferred because: +- GSSAPI/Kerberos requires the proxy to hold a service principal keytab and participate in the Kerberos infrastructure (KDC, realm trust, service tickets). This is a fundamentally different operational model from the credential store or JWKS endpoint approaches used by SCRAM and OAUTHBEARER. +- Terminating Kerberos at the proxy would require the proxy to impersonate the broker's service principal (or hold its own), raising complex delegation and trust questions. +- The demand for Kerberos termination (as opposed to passthrough) is lower than for SCRAM and OAUTHBEARER, which cover the most common credential isolation and identity provider integration use cases. +- If GSSAPI support is needed in the future, it could be added as a new `MechanismHandler` implementation, but the operational and trust model would need careful design. + ## References - [Proposal 004 — Terminology for Authentication][proposal-004] From e3a4bbb1b9a0a5a524ce026ee87c30aca05c344e Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 24 Jul 2026 05:20:38 +0000 Subject: [PATCH 20/52] docs(proposal): add component overview diagram Add a Mermaid class diagram showing the key types across the three implementation modules and their relationships: filter and mechanism handlers, credential store SPI, and keystore provider. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 96 +++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index e6e866e8..560c8c83 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -57,6 +57,102 @@ It also aims to be flexible, so as to allow other mechanisms to be supported in The proposal is organized per-component. Each component section covers its summary, API surfaces, configuration, threats and mitigations, and known limitations. +### Component overview + +The following diagram shows the key types across the three implementation modules and their relationships. Namespaces correspond to modules: **SaslTermination** → `kroxylicious-filters/kroxylicious-sasl-termination` (Components 1–4), **CredentialStoreSPI** → `kroxylicious-sasl-credential-store` (Component 5), **KeystoreProvider** → `kroxylicious-sasl-credential-store-provider-keystore` (Component 6). + +```mermaid +classDiagram + direction TB + + namespace SaslTermination { + class SaslTerminationFilter { + <> + } + class State { + <> + RequiringHandshake + RequiringAuthenticate + Authenticated + Failed + } + class MechanismHandlerFactory { + <> + +mechanismName() String + +initialize(MechanismConfig) + +createHandler() MechanismHandler + } + class MechanismHandler { + <> + +handleAuthenticate(byte[]) CompletionStage~AuthenticationResult~ + } + class AuthenticationResult { + <> + CHALLENGE / SUCCESS / FAILURE + } + class MechanismConfig { + <> + } + class ScramMechanismConfig + class OauthBearerMechanismConfig + class ScramSha256HandlerFactory + class ScramSha512HandlerFactory + class ScramHandler + class OauthBearerHandlerFactory + class OauthBearerHandler + } + + namespace CredentialStoreSPI { + class ScramCredentialStore { + <> + +lookupCredential(String) CompletionStage~ScramCredential~ + } + class ScramCredentialStoreService~C~ { + <> + +initialize(C) + +buildCredentialStore() ScramCredentialStore + } + class ScramCredential { + <> + } + } + + namespace KeystoreProvider { + class KeystoreScramCredentialStoreService { + <> + } + class KeystoreCredentialTool { + <> + } + } + + SaslTerminationFilter *-- State + SaslTerminationFilter ..> MechanismHandlerFactory : discovers via ServiceLoader + MechanismHandlerFactory --> MechanismHandler : creates per connection + MechanismHandler --> AuthenticationResult : returns + MechanismHandlerFactory ..> MechanismConfig : configured by + + ScramMechanismConfig ..|> MechanismConfig + OauthBearerMechanismConfig ..|> MechanismConfig + + ScramSha256HandlerFactory ..|> MechanismHandlerFactory + ScramSha512HandlerFactory ..|> MechanismHandlerFactory + OauthBearerHandlerFactory ..|> MechanismHandlerFactory + + ScramSha256HandlerFactory --> ScramHandler : creates + ScramSha512HandlerFactory --> ScramHandler : creates + OauthBearerHandlerFactory --> OauthBearerHandler : creates + + ScramHandler ..|> MechanismHandler + OauthBearerHandler ..|> MechanismHandler + + ScramHandler --> ScramCredentialStore : looks up credentials + ScramCredentialStore --> ScramCredential : returns + + ScramCredentialStoreService --> ScramCredentialStore : builds + KeystoreScramCredentialStoreService ..|> ScramCredentialStoreService +``` + ### Component 1: SaslTermination filter #### Summary From a1a7edbc5ba93e0b61e11f6194ba3ca596bd9f4f Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 27 Jul 2026 00:45:26 +0000 Subject: [PATCH 21/52] docs(proposal): address review feedback on SASL termination proposal - Clarify credential store is per-filter-instance, not global - Add component overview paragraph before architecture diagram - Add fixedAuthDelay config option for timing side-channel mitigation - Document MechanismHandler lifecycle (dispose on success/failure, not on connection close due to Filter API limitation) - Explain why there is no explicit Expired state in the state machine - Clarify API_VERSIONS is accepted in all states including expired - Add observability section with runtime and filter-specific metrics - Document file permission enforcement via env var for K8s/OpenShift Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 49 +++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 9 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 560c8c83..b6ea3b0b 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -44,7 +44,7 @@ In a zero-trust architecture the proxy can enforce authentication at the network ### Centralized credential management -A single credential store serves all proxy instances, rather than requiring per-broker credential configuration. Combined with the proxy's existing plugin system, this allows integration with enterprise credential stores. +The credential store is per-filter-instance. Clients of multiple brokers authenticate against a shared credential store, rather than requiring per-broker credential configuration. Combined with the proxy's existing plugin system, this allows integration with enterprise credential stores. ### Broker-less authentication @@ -59,6 +59,8 @@ The proposal is organized per-component. Each component section covers its summa ### Component overview +The `SaslTerminationFilter` (Component 1) intercepts SASL requests and manages per-connection authentication state via a sealed state machine. It delegates the actual authentication exchange to mechanism-specific `MechanismHandler` instances (Component 2), created per-connection by `MechanismHandlerFactory` implementations discovered via ServiceLoader. The OAUTHBEARER handler factory (Component 4) validates JWT tokens against a JWKS endpoint. The SCRAM handler factories (Component 3) use a `ScramCredentialStore` (Component 5) to look up stored credentials — a public SPI with a first-party KeyStore-backed provider (Component 6). + The following diagram shows the key types across the three implementation modules and their relationships. Namespaces correspond to modules: **SaslTermination** → `kroxylicious-filters/kroxylicious-sasl-termination` (Components 1–4), **CredentialStoreSPI** → `kroxylicious-sasl-credential-store` (Component 5), **KeystoreProvider** → `kroxylicious-sasl-credential-store-provider-keystore` (Component 6). ```mermaid @@ -181,6 +183,10 @@ The filter maintains per-connection state using a sealed interface `State` with | **Authenticated** | non-SASL request, session expired | reject and close | | **Failed** | *(terminal — connection closed)* | — | +**Why there is no `Expired` state:** Session expiry is a property of the `Authenticated` state, checked lazily when the next non-SASL request arrives. An explicit `Expired` state was considered but would be momentary — the connection is immediately either closed (non-SASL request) or transitions to `RequiringAuthenticate` (reauthentication handshake). It would also complicate the handshake guard, which currently accepts handshakes from `RequiringHandshake` and `Authenticated`. The expiry check is a simple conditional within `handleDefaultRequest`, which is easier to audit than an additional state with duplicated transition methods. + +**In-flight requests at expiry:** The filter checks expiry before forwarding, so the request that triggers the expiry check never reaches the broker. Previously-forwarded requests whose responses are still in flight will be delivered to the client before the connection closes. + - **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests, which negotiate the mechanism and transition to `RequiringAuthenticate`. - **RequiringAuthenticate:** Accepts `SASL_AUTHENTICATE` requests. Loops back to itself for multi-round mechanisms (e.g. SCRAM). Carries a reference to the `MechanismHandler` for the negotiated mechanism. - **Authenticated:** Success state. The filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters, then forwards all subsequent requests. If reauthentication is configured, this state also stores the session expiry time and allows transition back to `RequiringAuthenticate` via a new `SASL_HANDSHAKE`. @@ -202,7 +208,7 @@ Reauthentication is a protocol-level feature, not mechanism-specific — all mec **Client behaviour:** Standard Kafka clients (4.0+) handle reauthentication transparently via the `Selector`. When the session nears expiry, the client sends a new `SASL_HANDSHAKE` + `SASL_AUTHENTICATE` sequence over the existing connection. This is invisible to application code. -**Server-side enforcement:** If the session has expired and a non-SASL request arrives, the filter rejects it with `SASL_AUTHENTICATION_FAILED` and closes the connection. `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests are always accepted regardless of session expiry, to allow reauthentication. +**Server-side enforcement:** If the session has expired and a non-SASL request arrives, the filter rejects it with `SASL_AUTHENTICATION_FAILED` and closes the connection. `API_VERSIONS`, `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests are always accepted regardless of session expiry — `API_VERSIONS` is handled unconditionally before the state machine, and the SASL requests allow reauthentication. #### API surfaces @@ -221,6 +227,7 @@ The filter is configured via `SaslTerminationConfig`: |--------|------|----------|---------|-------------| | `mechanisms` | `Map` | Yes | -- | Map of IANA-registered mechanism name to mechanism-specific configuration. At least one entry is required. | | `maxTimeBeforeReauth` | `Duration` | No | disabled | Maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. | +| `fixedAuthDelay` | `Duration` | No | `200ms` | Fixed delay applied to all authentication rounds to prevent timing side-channel attacks that could enable user enumeration. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. | The `mechanisms` map values are polymorphic. Jackson deduction-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION)`) resolves the concrete type from the fields present: @@ -236,6 +243,7 @@ filters: - type: SaslTermination config: maxTimeBeforeReauth: 1h + fixedAuthDelay: 200ms mechanisms: SCRAM-SHA-256: credentialStore: KeystoreScramCredentialStoreService @@ -288,6 +296,10 @@ public interface MechanismHandler { } ``` +**`MechanismHandler` lifecycle:** The filter calls `dispose()` on the handler after SUCCESS (the handler is no longer needed once the client is authenticated) and after FAILURE (the connection is about to close). It is *not* called on raw connection close (e.g. client disconnects mid-exchange) because the `Filter` API has no connection-close hook — the handler becomes unreachable and is garbage collected. Handler implementations must therefore not hold resources that require explicit cleanup beyond what GC provides. + +For reauthentication (KIP-368), the previous handler was already disposed at SUCCESS time, so a fresh handler is created for the new exchange. + **`MechanismHandlerFactory`** -- manages mechanism-specific resources and creates handler instances. Discovered via `ServiceLoader`. ```java @@ -372,7 +384,7 @@ Key features: - **Multi-round SCRAM exchange.** SCRAM is a challenge-response protocol. The handler processes the client-first-message (round 1) and subsequent rounds, returning `CHALLENGE` until the exchange completes. - **Delegation to Kafka's `SaslServer`.** The handler does not reimplement SCRAM. It creates a Kafka `SaslServer` with a `CallbackHandler` that supplies the looked-up credential, then processes all messages through it. This benefits from Kafka's battle-tested implementation. -- **Timing side-channel mitigation.** A fixed delay is applied to all authentication rounds to prevent attackers from distinguishing existing from non-existing users by measuring response times. +- **Timing side-channel mitigation.** A configurable fixed delay (`fixedAuthDelay`) is applied to all authentication rounds to prevent attackers from distinguishing existing from non-existing users by measuring response times. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. #### Authentication flow @@ -409,7 +421,7 @@ public record ScramMechanismConfig( | Threat | Mitigation | |--------|------------| | Username enumeration -- an attacker distinguishes existing from non-existing users by observing different error messages. | When a user is not found, the handler returns a generic `"Authentication failed"` error message identical to the message returned for incorrect credentials. | -| Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the handler applies a fixed delay to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. | +| Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter applies a configurable fixed delay (`fixedAuthDelay`) to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. | | SCRAM protocol correctness -- a bug in the SCRAM implementation could allow authentication bypass or credential leakage. | Delegated to Kafka's own `SaslServer`, which is widely deployed and well-tested. The handler is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. | #### Known limitations @@ -685,7 +697,7 @@ List all usernames in the KeyStore. | Threat | Mitigation | |--------|------------| -| KeyStore file exposure -- an attacker gains read access to the KeyStore file on disk. | POSIX file permission check: the provider refuses to load a KeyStore with group or world read/write permissions. The KeyStore itself is password-encrypted. | +| KeyStore file exposure -- an attacker gains read access to the KeyStore file on disk. | POSIX file permission check: the provider checks file permissions before loading, requiring `0600` or stricter by default. On Kubernetes/OpenShift where group-readable files are necessary, the `KROXYLICIOUS_DANGEROUSLY_CHANGE_PERMISSION_CHECK` environment variable allows relaxing to `0640`. The KeyStore itself is password-encrypted. | **Accepted risk: credential material in JVM heap.** SCRAM credential data (serverKey, storedKey, salt) is held in memory for the lifetime of the proxy. An attacker who can obtain a heap dump (e.g. via JMX, `/proc//mem`, or a core dump) can extract this material. There is no practical mitigation within a JVM. Operators should protect heap dump access through operational controls (JMX authentication, file permissions on core dumps, container security policies). @@ -713,7 +725,7 @@ The implementation is organized into three modules, following the same pattern a - **KeyStore encryption:** Credentials are stored in Java KeyStore files, encrypted with the KeyStore password. File-system permissions and KeyStore passwords are the primary access controls. - **PasswordProvider abstraction:** Production deployments should use file-based passwords rather than inline passwords in configuration. The `PasswordProvider` interface supports both. -- **File permission enforcement:** On POSIX systems, the credential store refuses to load a KeyStore file that has group or world read/write permissions. This prevents accidental exposure of credential material through overly permissive file modes. +- **File permission enforcement:** On POSIX systems, the credential store checks the KeyStore file's permissions before loading it. By default, group or world read/write permissions are rejected (`0600` or stricter required). This prevents accidental exposure of credential material through overly permissive file modes. The required permission level is configurable via the `KROXYLICIOUS_DANGEROUSLY_CHANGE_PERMISSION_CHECK` environment variable, which can be set to `0640` to allow group-readable files. This is necessary on OpenShift, where the `restricted-v2` SCC runs containers as an arbitrary UID while Secret volume files are owned by root — requiring group-readable permissions (`defaultMode: 0440` with `fsGroup`) for the container process to access them. The environment variable is set in the PodSpec by the `kroxylicious-operator`, keeping the trust chain secure: operator → pod spec → env var → policy, with no writable config file in the loop. Using a config file for this setting would create a bootstrapping problem — if the config file itself were group-writable, an attacker could downgrade the permission policy. - **In-memory handling:** `ScramCredential` uses defensive copies for `byte[]` fields (correctness measure against accidental mutation) and `toString()` redacts sensitive fields (prevents log leakage). Credential material in the JVM heap is an accepted risk — see Component 6 threat discussion. ### SCRAM protocol correctness @@ -726,7 +738,26 @@ When a user is not found in the credential store, the handler returns a generic ### Timing side-channel mitigation -Without mitigation, an attacker could distinguish existing from non-existing users by measuring response times: credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists. Rather than trying to equalize these inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the SCRAM handler applies a fixed delay to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. +Without mitigation, an attacker could distinguish existing from non-existing users by measuring response times: credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists. Rather than trying to equalize these inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter applies a configurable fixed delay (`fixedAuthDelay`) to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. + +### Observability + +#### Runtime-level metrics + +The proxy runtime emits authentication outcome metrics when any filter announces an authentication result via `FilterContext.clientSaslAuthenticationSuccess()` or `clientSaslAuthenticationFailure()`. These apply uniformly to all authentication approaches (SASL termination, SASL inspection, transport authentication). + +| Metric | Type | Tags | Description | +|--------|------|------|-------------| +| `kroxylicious_client_auth_total` | Counter | `virtual_cluster`, `mechanism`, `outcome` (`success` / `failure`) | Authentication outcomes. | + +#### Filter-specific metrics + +The SASL termination filter emits additional metrics for authentication latency and session expiry. + +| Metric | Type | Tags | Description | +|--------|------|------|-------------| +| `kroxylicious_filter_sasl_termination_auth_duration_seconds` | Timer | `virtual_cluster`, `mechanism` | Authentication latency, exclusive of the configured fixed timing delay. Measures the real work: credential store lookup, token validation, SCRAM rounds. | +| `kroxylicious_filter_sasl_termination_session_expired_total` | Counter | `virtual_cluster`, `mechanism` | Sessions that expired without the client reauthenticating. | ### Connection lifecycle safety @@ -759,9 +790,9 @@ Without mitigation, an attacker could distinguish existing from non-existing use **Not affected:** - `kroxylicious-api` — no API changes needed (uses existing `clientSaslAuthenticationSuccess`/`clientSaslAuthenticationFailure` from proposal 006) -- `kroxylicious-runtime` — no runtime changes +- `kroxylicious-runtime` — authentication outcome metrics (`kroxylicious_client_auth_total`) - `kroxylicious-kms` and KMS providers — unrelated -- `kroxylicious-kubernetes` — no operator changes (the termination filter is configured via standard filter configuration) +- `kroxylicious-kubernetes` — the operator should set `KROXYLICIOUS_DANGEROUSLY_CHANGE_PERMISSION_CHECK=0640` in the PodSpec on OpenShift (and optionally on plain Kubernetes) to allow group-readable Secret volume mounts ## Compatibility From c159e17736c2f25c32149cd574d1851e3bcd14ae Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 27 Jul 2026 05:10:38 +0000 Subject: [PATCH 22/52] docs(proposal): replace monolithic diagram with focused views Split the single large class diagram into three complementary views: - Module dependency graph showing the three modules - Pruned class diagram showing only SPI contract types - State machine diagram for the per-connection authentication states Removes implementation classes (handler factories, concrete handlers, config subtypes) from the diagram since they are already described in their respective component sections. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 55 +++++++++++++++---------------- 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index b6ea3b0b..41336595 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -61,7 +61,19 @@ The proposal is organized per-component. Each component section covers its summa The `SaslTerminationFilter` (Component 1) intercepts SASL requests and manages per-connection authentication state via a sealed state machine. It delegates the actual authentication exchange to mechanism-specific `MechanismHandler` instances (Component 2), created per-connection by `MechanismHandlerFactory` implementations discovered via ServiceLoader. The OAUTHBEARER handler factory (Component 4) validates JWT tokens against a JWKS endpoint. The SCRAM handler factories (Component 3) use a `ScramCredentialStore` (Component 5) to look up stored credentials — a public SPI with a first-party KeyStore-backed provider (Component 6). -The following diagram shows the key types across the three implementation modules and their relationships. Namespaces correspond to modules: **SaslTermination** → `kroxylicious-filters/kroxylicious-sasl-termination` (Components 1–4), **CredentialStoreSPI** → `kroxylicious-sasl-credential-store` (Component 5), **KeystoreProvider** → `kroxylicious-sasl-credential-store-provider-keystore` (Component 6). +The implementation spans three modules. Their dependencies: + +```mermaid +graph LR + ST["kroxylicious-sasl-termination
(Components 1–4)"] + CS["kroxylicious-sasl-credential-store
(Component 5)"] + KP["kroxylicious-sasl-credential-store-provider-keystore
(Component 6)"] + + ST --> CS + KP --> CS +``` + +The key types and their relationships across these modules: ```mermaid classDiagram @@ -95,13 +107,6 @@ classDiagram class MechanismConfig { <> } - class ScramMechanismConfig - class OauthBearerMechanismConfig - class ScramSha256HandlerFactory - class ScramSha512HandlerFactory - class ScramHandler - class OauthBearerHandlerFactory - class OauthBearerHandler } namespace CredentialStoreSPI { @@ -123,9 +128,6 @@ classDiagram class KeystoreScramCredentialStoreService { <> } - class KeystoreCredentialTool { - <> - } } SaslTerminationFilter *-- State @@ -133,25 +135,10 @@ classDiagram MechanismHandlerFactory --> MechanismHandler : creates per connection MechanismHandler --> AuthenticationResult : returns MechanismHandlerFactory ..> MechanismConfig : configured by - - ScramMechanismConfig ..|> MechanismConfig - OauthBearerMechanismConfig ..|> MechanismConfig - - ScramSha256HandlerFactory ..|> MechanismHandlerFactory - ScramSha512HandlerFactory ..|> MechanismHandlerFactory - OauthBearerHandlerFactory ..|> MechanismHandlerFactory - - ScramSha256HandlerFactory --> ScramHandler : creates - ScramSha512HandlerFactory --> ScramHandler : creates - OauthBearerHandlerFactory --> OauthBearerHandler : creates - - ScramHandler ..|> MechanismHandler - OauthBearerHandler ..|> MechanismHandler - - ScramHandler --> ScramCredentialStore : looks up credentials - ScramCredentialStore --> ScramCredential : returns + MechanismHandler ..> ScramCredentialStore : looks up credentials (SCRAM) ScramCredentialStoreService --> ScramCredentialStore : builds + ScramCredentialStore --> ScramCredential : returns KeystoreScramCredentialStoreService ..|> ScramCredentialStoreService ``` @@ -183,6 +170,18 @@ The filter maintains per-connection state using a sealed interface `State` with | **Authenticated** | non-SASL request, session expired | reject and close | | **Failed** | *(terminal — connection closed)* | — | +```mermaid +stateDiagram-v2 + [*] --> RequiringHandshake + RequiringHandshake --> RequiringAuthenticate : SASL_HANDSHAKE (supported mechanism) + RequiringAuthenticate --> RequiringAuthenticate : CHALLENGE + RequiringAuthenticate --> Authenticated : SUCCESS + RequiringAuthenticate --> Failed : FAILURE + Authenticated --> RequiringAuthenticate : SASL_HANDSHAKE (reauthentication) + Authenticated --> [*] : session expired + Failed --> [*] +``` + **Why there is no `Expired` state:** Session expiry is a property of the `Authenticated` state, checked lazily when the next non-SASL request arrives. An explicit `Expired` state was considered but would be momentary — the connection is immediately either closed (non-SASL request) or transitions to `RequiringAuthenticate` (reauthentication handshake). It would also complicate the handshake guard, which currently accepts handshakes from `RequiringHandshake` and `Authenticated`. The expiry check is a simple conditional within `handleDefaultRequest`, which is easier to audit than an additional state with duplicated transition methods. **In-flight requests at expiry:** The filter checks expiry before forwarding, so the request that triggers the expiry check never reaches the broker. Previously-forwarded requests whose responses are still in flight will be delivered to the client before the connection closes. From aef3f2293ca5df3dac8f547e9c226948dc7811af Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 27 Jul 2026 05:21:52 +0000 Subject: [PATCH 23/52] docs(proposal): fix state diagram rendering Replace self-loop with a note to avoid overlapping edges, switch to LR layout, and shorten transition labels. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 41336595..f20c6e14 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -172,12 +172,13 @@ The filter maintains per-connection state using a sealed interface `State` with ```mermaid stateDiagram-v2 + direction LR [*] --> RequiringHandshake - RequiringHandshake --> RequiringAuthenticate : SASL_HANDSHAKE (supported mechanism) - RequiringAuthenticate --> RequiringAuthenticate : CHALLENGE + RequiringHandshake --> RequiringAuthenticate : supported mechanism + note right of RequiringAuthenticate : Loops on CHALLENGE RequiringAuthenticate --> Authenticated : SUCCESS RequiringAuthenticate --> Failed : FAILURE - Authenticated --> RequiringAuthenticate : SASL_HANDSHAKE (reauthentication) + Authenticated --> RequiringAuthenticate : reauthentication Authenticated --> [*] : session expired Failed --> [*] ``` From 0973fba7ebdaf4f35dc43426ccacdde283a0df21 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 01:40:46 +0000 Subject: [PATCH 24/52] docs(proposal): add delegation tokens as out of scope Delegation tokens are not supported with SASL termination because token credentials are broker-managed state. The filter removes delegation token APIs from API_VERSIONS and rejects those requests. Notes future support may be possible via DescribeDelegationToken. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index f20c6e14..ddd17577 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -268,6 +268,7 @@ filters: #### Known limitations - The filter does not support SASL PLAIN or GSSAPI (Kerberos). See [Rejected alternatives](#rejected-alternatives). +- **Delegation tokens are not supported.** The filter removes the delegation token APIs (`CreateDelegationToken`, `RenewDelegationToken`, `ExpireDelegationToken`, `DescribeDelegationToken`) from the `API_VERSIONS` response and rejects those request types with a clear error. Future support may be possible by using `DescribeDelegationToken` to sync token credentials from the broker into the proxy's credential store. --- From 337e69788bf76c3d3b9a9ade853db4502e4e8ab2 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 01:41:40 +0000 Subject: [PATCH 25/52] docs(proposal): acknowledge SCRAM credential isolation trade-offs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Existing Kafka credential management tooling cannot be used with the proxy's credential store, and existing broker SCRAM users cannot be migrated — they must be re-provisioned from scratch. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index ddd17577..b25831f4 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -30,6 +30,8 @@ With SASL termination, the proxy authenticates clients using credentials stored - Different credential lifecycles are needed for client-facing and broker-facing authentication. - Compliance requirements mandate credential isolation between organizational boundaries. +**Operational trade-off for SCRAM:** Because the proxy maintains its own credential store, existing Kafka credential management tooling (`KafkaUser` CRs, `kafka-configs.sh`, the Admin API's `AlterUserScramCredentials`) cannot be used to manage proxy credentials. Operators with existing SCRAM users on their brokers cannot migrate those credentials to the proxy — they must re-provision users in the proxy's credential store from scratch. + ### Authentication protocol translation The proxy can authenticate clients using one SASL mechanism (e.g. `SCRAM-SHA-256`) while using an entirely different authentication mechanism to connect to the broker (e.g. mTLS, or `OAUTHBEARER`). This enables: From 5534b83f883e41c5de8191487eb1374d6aad436e Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 01:44:30 +0000 Subject: [PATCH 26/52] docs(proposal): remove SCRAM credential APIs from API_VERSIONS AlterUserScramCredentials and DescribeUserScramCredentials operate on the broker's credential store, which is irrelevant when the proxy terminates SASL. The filter removes these APIs (apiKeys 50, 51) from the API_VERSIONS response and rejects those request types. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index b25831f4..ef03ca73 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -30,7 +30,7 @@ With SASL termination, the proxy authenticates clients using credentials stored - Different credential lifecycles are needed for client-facing and broker-facing authentication. - Compliance requirements mandate credential isolation between organizational boundaries. -**Operational trade-off for SCRAM:** Because the proxy maintains its own credential store, existing Kafka credential management tooling (`KafkaUser` CRs, `kafka-configs.sh`, the Admin API's `AlterUserScramCredentials`) cannot be used to manage proxy credentials. Operators with existing SCRAM users on their brokers cannot migrate those credentials to the proxy — they must re-provision users in the proxy's credential store from scratch. +**Operational trade-off for SCRAM:** Because the proxy maintains its own credential store, existing Kafka credential management tooling (`KafkaUser` CRs, `kafka-configs.sh`, the Admin API's `AlterUserScramCredentials` / `DescribeUserScramCredentials`) cannot be used to manage proxy credentials. These APIs operate on the broker's credential store, which is irrelevant when the proxy terminates SASL. The filter removes these APIs (apiKeys 50, 51) from the `API_VERSIONS` response and rejects those request types with a clear error. Operators with existing SCRAM users on their brokers cannot migrate those credentials to the proxy — they must re-provision users in the proxy's credential store from scratch. ### Authentication protocol translation From 29aef91a885ff6ee6ba14757a59a7895db666278 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 01:46:35 +0000 Subject: [PATCH 27/52] docs(proposal): distinguish Alter vs Describe SCRAM credential APIs AlterUserScramCredentials is rejected (credential store SPI is read-only). DescribeUserScramCredentials can be answered by the filter from its own credential store since the response contains only mechanism and iteration count. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index ef03ca73..c3235ba9 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -30,7 +30,7 @@ With SASL termination, the proxy authenticates clients using credentials stored - Different credential lifecycles are needed for client-facing and broker-facing authentication. - Compliance requirements mandate credential isolation between organizational boundaries. -**Operational trade-off for SCRAM:** Because the proxy maintains its own credential store, existing Kafka credential management tooling (`KafkaUser` CRs, `kafka-configs.sh`, the Admin API's `AlterUserScramCredentials` / `DescribeUserScramCredentials`) cannot be used to manage proxy credentials. These APIs operate on the broker's credential store, which is irrelevant when the proxy terminates SASL. The filter removes these APIs (apiKeys 50, 51) from the `API_VERSIONS` response and rejects those request types with a clear error. Operators with existing SCRAM users on their brokers cannot migrate those credentials to the proxy — they must re-provision users in the proxy's credential store from scratch. +**Operational trade-off for SCRAM:** Because the proxy maintains its own credential store, existing Kafka credential management tooling (`KafkaUser` CRs, `kafka-configs.sh`, the Admin API's `AlterUserScramCredentials`) cannot be used to manage proxy credentials. The filter removes `AlterUserScramCredentials` (apiKey 51) from the `API_VERSIONS` response and rejects those requests with a clear error, since the credential store SPI is read-only. `DescribeUserScramCredentials` (apiKey 50) can be answered by the filter from its own credential store — the response contains only the mechanism and iteration count, no sensitive material. Operators with existing SCRAM users on their brokers cannot migrate those credentials to the proxy — they must re-provision users in the proxy's credential store from scratch. ### Authentication protocol translation From d19a2ae00bba039f6c93a591fbd55c080f3bcfbe Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 01:56:14 +0000 Subject: [PATCH 28/52] docs(proposal): acknowledge upstream auth failure visibility gap SASL termination splits authentication into two independent exchanges. The client can succeed against the proxy while the proxy fails to authenticate upstream. Discusses potential solutions (triggering upstream auth verification, SaslInitiator filter) but defers to future work due to complexity. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index c3235ba9..6b299208 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -271,6 +271,7 @@ filters: - The filter does not support SASL PLAIN or GSSAPI (Kerberos). See [Rejected alternatives](#rejected-alternatives). - **Delegation tokens are not supported.** The filter removes the delegation token APIs (`CreateDelegationToken`, `RenewDelegationToken`, `ExpireDelegationToken`, `DescribeDelegationToken`) from the `API_VERSIONS` response and rejects those request types with a clear error. Future support may be possible by using `DescribeDelegationToken` to sync token credentials from the broker into the proxy's credential store. +- **Upstream authentication failure is not surfaced to the client.** SASL termination splits authentication into two independent exchanges: client-to-proxy and proxy-to-broker. The client can authenticate successfully against the proxy's credential store, but the proxy's own authentication to the broker may fail independently (wrong credentials, expired certificates, misconfigured mTLS). In this case the client has already been told authentication succeeded, and will only discover the problem when subsequent requests fail with broker-level errors. Ideally the filter would verify upstream authentication before reporting success to the client — for example, by triggering an internal request to force authentication on the broker connection. However, this is complex: the filter would need to act as both a SASL terminator and a SASL initiator (or coordinate with a separate `SaslInitiator` filter), and the broker connection may not even use SASL (e.g. mTLS). Operators can diagnose the disconnect via logging and metrics, which distinguish client-side and broker-side authentication outcomes. A full solution is deferred to future work. --- From 4ee66d603a3afae7a09a60859af815c4fa91412f Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 01:57:44 +0000 Subject: [PATCH 29/52] docs(proposal): move upstream auth failure to open questions Upstream auth failure visibility is an open design question, not a known limitation. Moved to a new Open Questions section describing the problem and potential approaches (SaslInitiator filter, indistinguishable failure responses) without making a decision. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 6b299208..77370ba7 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -271,7 +271,6 @@ filters: - The filter does not support SASL PLAIN or GSSAPI (Kerberos). See [Rejected alternatives](#rejected-alternatives). - **Delegation tokens are not supported.** The filter removes the delegation token APIs (`CreateDelegationToken`, `RenewDelegationToken`, `ExpireDelegationToken`, `DescribeDelegationToken`) from the `API_VERSIONS` response and rejects those request types with a clear error. Future support may be possible by using `DescribeDelegationToken` to sync token credentials from the broker into the proxy's credential store. -- **Upstream authentication failure is not surfaced to the client.** SASL termination splits authentication into two independent exchanges: client-to-proxy and proxy-to-broker. The client can authenticate successfully against the proxy's credential store, but the proxy's own authentication to the broker may fail independently (wrong credentials, expired certificates, misconfigured mTLS). In this case the client has already been told authentication succeeded, and will only discover the problem when subsequent requests fail with broker-level errors. Ideally the filter would verify upstream authentication before reporting success to the client — for example, by triggering an internal request to force authentication on the broker connection. However, this is complex: the filter would need to act as both a SASL terminator and a SASL initiator (or coordinate with a separate `SaslInitiator` filter), and the broker connection may not even use SASL (e.g. mTLS). Operators can diagnose the disconnect via logging and metrics, which distinguish client-side and broker-side authentication outcomes. A full solution is deferred to future work. --- @@ -825,6 +824,14 @@ The `KeystoreCredentialManager` class does expose `ScramMechanism` in its public [kafka-javadoc]: https://kafka.apache.org/43/javadoc/index.html [proposal-116]: https://github.com/kroxylicious/design/pull/116 +## Open questions + +### Upstream authentication failure visibility + +SASL termination splits authentication into two independent exchanges: client-to-proxy and proxy-to-broker. The client can authenticate successfully against the proxy's credential store, but the proxy's own authentication to the broker may fail independently (wrong credentials, expired certificates, misconfigured mTLS). In this case the client has already been told authentication succeeded, and will only discover the problem when subsequent requests fail with broker-level errors. + +Ideally the filter would verify upstream authentication before reporting success to the client — for example, by triggering an internal request to force authentication on the broker connection. However, this is complex: the filter would need to act as both a SASL terminator and a SASL initiator (or coordinate with a separate `SaslInitiator` filter), and the broker connection may not even use SASL (e.g. mTLS). The failure response to the client should be indistinguishable from a terminated auth failure, so that an attacker cannot determine which side failed. Logging and metrics would help operators distinguish the two cases. + ## Rejected alternatives ### Generic CredentialStore covering all mechanisms From b09d8d06ee451eecd8479b2fce9de2fcd739bb0a Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:00:10 +0000 Subject: [PATCH 30/52] docs(proposal): clarify filter intercepts all requests The filter intercepts all requests, not just SASL ones. Non-SASL requests are checked for authentication state and session expiry as a security barrier. The unauthenticated check is defence in depth since failed auth already closes the connection. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 77370ba7..e7d752ad 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -148,7 +148,7 @@ classDiagram #### Summary -The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, authenticating clients at the proxy and short-circuiting the responses without forwarding them to the broker. Multiple mechanisms are configured within a single filter instance because the Kafka SASL protocol requires it: the client sends a `SaslHandshakeRequest` naming its chosen mechanism, and the server responds with the set of supported mechanisms. A filter-per-mechanism model would not work because no single filter would have the complete set of supported mechanisms to advertise in the `SaslHandshakeResponse`. +The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts all requests on a connection. For `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, it authenticates clients at the proxy, short-circuiting those exchanges without forwarding them to the broker. For all other request types, the filter enforces the security barrier: if the client has not completed authentication, the request is rejected and the connection is closed; if the session lifetime has elapsed, the filter does the same, requiring the client to reauthenticate. In practice the unauthenticated rejection should never be reached for non-SASL requests, because failed authentication already closes the connection — but the check exists for defence in depth. Multiple mechanisms are configured within a single filter instance because the Kafka SASL protocol requires it: the client sends a `SaslHandshakeRequest` naming its chosen mechanism, and the server responds with the set of supported mechanisms. A filter-per-mechanism model would not work because no single filter would have the complete set of supported mechanisms to advertise in the `SaslHandshakeResponse`. Key features: From fd616eed619c005e53d4eecf7fe2e8c63bf3ba54 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:00:57 +0000 Subject: [PATCH 31/52] docs(proposal): fail closed on unknown SASL request versions Reject and close the connection if SASL_HANDSHAKE or SASL_AUTHENTICATE arrives with an API version outside the range known to the filter, preventing future protocol versions from bypassing security logic. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index e7d752ad..f65dc8c6 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -153,6 +153,7 @@ The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that inter Key features: - **Security barrier.** Until a client has successfully authenticated, the only requests permitted are `API_VERSIONS`, `SASL_HANDSHAKE`, and `SASL_AUTHENTICATE`. All other request types are rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. +- **Fail closed on unknown versions.** If a `SASL_HANDSHAKE` or `SASL_AUTHENTICATE` request arrives with an API version outside the range known to the filter, the filter rejects the request and closes the connection. This prevents a future protocol version from bypassing the filter's security logic. - **State machine.** Per-connection authentication state is modelled as a sealed interface with four concrete states, preventing invalid transitions at compile time. - **Reauthentication (KIP-368).** When `maxTimeBeforeReauth` is configured, the filter includes a `sessionLifetimeMs` value in `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. Sessions that expire without reauthentication are rejected and closed. - **Mechanism dispatch.** The filter delegates each authentication exchange to a `MechanismHandler` obtained from the appropriate `MechanismHandlerFactory` (see Component 2). The filter itself is mechanism-agnostic. From 496041a61462fd319bfc08d3dd63bc330b07bf79 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:03:08 +0000 Subject: [PATCH 32/52] docs(proposal): add configurable SaslSubjectBuilderService Add optional subjectBuilder config option defaulting to DEFAULT_SUBJECT_BUILDER, consistent with SaslInspection filter. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index f65dc8c6..6a6f4cda 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -231,6 +231,7 @@ The filter is configured via `SaslTerminationConfig`: | `mechanisms` | `Map` | Yes | -- | Map of IANA-registered mechanism name to mechanism-specific configuration. At least one entry is required. | | `maxTimeBeforeReauth` | `Duration` | No | disabled | Maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. | | `fixedAuthDelay` | `Duration` | No | `200ms` | Fixed delay applied to all authentication rounds to prevent timing side-channel attacks that could enable user enumeration. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. | +| `subjectBuilder` | `SaslSubjectBuilderService` | No | `DEFAULT_SUBJECT_BUILDER` | Plugin for constructing the `Subject` from authentication results. Defaults to `DEFAULT_SUBJECT_BUILDER`, consistent with the existing SASL inspection filter. | The `mechanisms` map values are polymorphic. Jackson deduction-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION)`) resolves the concrete type from the fields present: From 693e3da4dfc2b07adbe4f4a786f16f50d0c9b9c9 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:09:30 +0000 Subject: [PATCH 33/52] docs(proposal): change mechanisms from map to list with type discriminator Replace Map with List using Jackson name-based polymorphism on a 'mechanism' property containing the IANA-registered mechanism name. Gives better error messages and is consistent with other definition lists in the project. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 6a6f4cda..ec8975c1 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -228,17 +228,12 @@ The filter is configured via `SaslTerminationConfig`: | Option | Type | Required | Default | Description | |--------|------|----------|---------|-------------| -| `mechanisms` | `Map` | Yes | -- | Map of IANA-registered mechanism name to mechanism-specific configuration. At least one entry is required. | +| `mechanisms` | `List` | Yes | -- | List of mechanism configurations. Each entry includes a `mechanism` field (the IANA-registered mechanism name) and mechanism-specific configuration. At least one entry is required. | | `maxTimeBeforeReauth` | `Duration` | No | disabled | Maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. | | `fixedAuthDelay` | `Duration` | No | `200ms` | Fixed delay applied to all authentication rounds to prevent timing side-channel attacks that could enable user enumeration. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. | | `subjectBuilder` | `SaslSubjectBuilderService` | No | `DEFAULT_SUBJECT_BUILDER` | Plugin for constructing the `Subject` from authentication results. Defaults to `DEFAULT_SUBJECT_BUILDER`, consistent with the existing SASL inspection filter. | -The `mechanisms` map values are polymorphic. Jackson deduction-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION)`) resolves the concrete type from the fields present: - -- If the entry contains `credentialStore` and `credentialStoreConfig`, it deserializes as `ScramMechanismConfig`. -- If the entry contains `jwksEndpointUrl`, `expectedAudience`, and `expectedIssuer`, it deserializes as `OauthBearerMechanismConfig`. - -This means the mechanism map key (e.g. `SCRAM-SHA-256`) selects which `MechanismHandlerFactory` handles the exchange, while the value's field structure determines which config type Jackson produces. There is no explicit type discriminator field. +The `mechanisms` list elements are polymorphic. Jackson name-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism")`) resolves the concrete type from the `mechanism` field, which is the IANA-registered mechanism name (e.g. `SCRAM-SHA-256`, `OAUTHBEARER`). This also selects which `MechanismHandlerFactory` handles the exchange. **Example configuration:** @@ -249,14 +244,14 @@ filters: maxTimeBeforeReauth: 1h fixedAuthDelay: 200ms mechanisms: - SCRAM-SHA-256: + - mechanism: SCRAM-SHA-256 credentialStore: KeystoreScramCredentialStoreService credentialStoreConfig: file: /path/to/credentials.p12 storePassword: file: /etc/kroxylicious/keystore-password.txt storeType: PKCS12 - OAUTHBEARER: + - mechanism: OAUTHBEARER jwksEndpointUrl: https://idp.example.com/.well-known/jwks.json expectedAudience: kafka expectedIssuer: https://idp.example.com @@ -348,13 +343,14 @@ public record AuthenticationResult( } ``` -**`MechanismConfig`** -- sealed interface for mechanism-specific configuration, using Jackson deduction-based polymorphism: +**`MechanismConfig`** -- sealed interface for mechanism-specific configuration, using Jackson name-based polymorphism on the `mechanism` field: ```java -@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION) +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism") @JsonSubTypes({ - @JsonSubTypes.Type(ScramMechanismConfig.class), - @JsonSubTypes.Type(OauthBearerMechanismConfig.class) + @JsonSubTypes.Type(value = ScramMechanismConfig.class, name = "SCRAM-SHA-256"), + @JsonSubTypes.Type(value = ScramMechanismConfig.class, name = "SCRAM-SHA-512"), + @JsonSubTypes.Type(value = OauthBearerMechanismConfig.class, name = "OAUTHBEARER") }) public sealed interface MechanismConfig permits ScramMechanismConfig, OauthBearerMechanismConfig { From 5a67abcf5aff1e44bc5c9bf910d4b54fa41c666a Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:13:21 +0000 Subject: [PATCH 34/52] docs(proposal): use per-variant SCRAM config subclasses ScramMechanismConfig becomes an abstract sealed base class whose constructor accepts the mechanism name. ScramSha256MechanismConfig and ScramSha512MechanismConfig are trivial subclasses, allowing Jackson name-based polymorphism to distinguish the two variants. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 47 +++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index ec8975c1..6457a1c7 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -348,8 +348,8 @@ public record AuthenticationResult( ```java @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism") @JsonSubTypes({ - @JsonSubTypes.Type(value = ScramMechanismConfig.class, name = "SCRAM-SHA-256"), - @JsonSubTypes.Type(value = ScramMechanismConfig.class, name = "SCRAM-SHA-512"), + @JsonSubTypes.Type(value = ScramSha256MechanismConfig.class, name = "SCRAM-SHA-256"), + @JsonSubTypes.Type(value = ScramSha512MechanismConfig.class, name = "SCRAM-SHA-512"), @JsonSubTypes.Type(value = OauthBearerMechanismConfig.class, name = "OAUTHBEARER") }) public sealed interface MechanismConfig @@ -357,6 +357,30 @@ public sealed interface MechanismConfig } ``` +`ScramMechanismConfig` is an abstract base class whose constructor accepts the mechanism name. The per-variant subclasses contain only a default constructor: + +```java +public abstract sealed class ScramMechanismConfig implements MechanismConfig + permits ScramSha256MechanismConfig, ScramSha512MechanismConfig { + + private final String mechanism; + + protected ScramMechanismConfig(String mechanism) { + this.mechanism = mechanism; + } + + // credentialStore, credentialStoreConfig fields... +} + +public final class ScramSha256MechanismConfig extends ScramMechanismConfig { + public ScramSha256MechanismConfig() { super("SCRAM-SHA-256"); } +} + +public final class ScramSha512MechanismConfig extends ScramMechanismConfig { + public ScramSha512MechanismConfig() { super("SCRAM-SHA-512"); } +} +``` + #### ServiceLoader discovery Factories are registered in `META-INF/services/io.kroxylicious.filter.sasl.termination.mechanism.MechanismHandlerFactory`. At filter factory initialization time, the `SaslTermination` filter factory loads all registered factories, matches them to the mechanism names present in the user's configuration, and calls `initialize()` on each matched factory. @@ -401,15 +425,20 @@ The SCRAM handler factories use: #### Configuration -SCRAM mechanisms are configured via `ScramMechanismConfig`: +SCRAM mechanisms are configured via `ScramMechanismConfig` (see Component 2 for the `ScramSha256MechanismConfig` / `ScramSha512MechanismConfig` subclasses). The base class carries the credential store configuration: ```java -public record ScramMechanismConfig( - @JsonProperty(required = true) - @PluginImplName(ScramCredentialStoreService.class) String credentialStore, - @JsonProperty(required = true) - @PluginImplConfig(implNameProperty = "credentialStore") Object credentialStoreConfig) - implements MechanismConfig { } +public abstract sealed class ScramMechanismConfig implements MechanismConfig + permits ScramSha256MechanismConfig, ScramSha512MechanismConfig { + + @JsonProperty(required = true) + @PluginImplName(ScramCredentialStoreService.class) + private String credentialStore; + + @JsonProperty(required = true) + @PluginImplConfig(implNameProperty = "credentialStore") + private Object credentialStoreConfig; +} ``` | Option | Type | Required | Default | Description | From fc3c7edec841b52fd229d6a74971825f07310024 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:14:40 +0000 Subject: [PATCH 35/52] docs(proposal): add shouldHandleRequest optimization Once authenticated with no session expiry, the filter uses shouldHandleRequest to skip deserialization of subsequent requests, avoiding unnecessary filter overhead in steady state. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 6457a1c7..ddff01b4 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -157,6 +157,7 @@ Key features: - **State machine.** Per-connection authentication state is modelled as a sealed interface with four concrete states, preventing invalid transitions at compile time. - **Reauthentication (KIP-368).** When `maxTimeBeforeReauth` is configured, the filter includes a `sessionLifetimeMs` value in `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. Sessions that expire without reauthentication are rejected and closed. - **Mechanism dispatch.** The filter delegates each authentication exchange to a `MechanismHandler` obtained from the appropriate `MechanismHandlerFactory` (see Component 2). The filter itself is mechanism-agnostic. +- **Steady-state optimization.** Once a client is in the `Authenticated` state with no session expiry configured, the filter uses `shouldHandleRequest` to avoid deserializing subsequent requests, letting them pass through without filter overhead. #### State machine From 29f29bfc0e8601b32d629f141bf2a720b52d6d69 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:21:08 +0000 Subject: [PATCH 36/52] docs(proposal): confirm Kafka SaslServer impls are GC-safe Note that Kafka's ScramSaslServer.dispose() is a no-op, confirming that the lack of dispose() on raw connection close is safe for the built-in handlers. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index ddff01b4..91627f32 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -297,7 +297,7 @@ public interface MechanismHandler { } ``` -**`MechanismHandler` lifecycle:** The filter calls `dispose()` on the handler after SUCCESS (the handler is no longer needed once the client is authenticated) and after FAILURE (the connection is about to close). It is *not* called on raw connection close (e.g. client disconnects mid-exchange) because the `Filter` API has no connection-close hook — the handler becomes unreachable and is garbage collected. Handler implementations must therefore not hold resources that require explicit cleanup beyond what GC provides. +**`MechanismHandler` lifecycle:** The filter calls `dispose()` on the handler after SUCCESS (the handler is no longer needed once the client is authenticated) and after FAILURE (the connection is about to close). It is *not* called on raw connection close (e.g. client disconnects mid-exchange) because the `Filter` API has no connection-close hook — the handler becomes unreachable and is garbage collected. Handler implementations must therefore not hold resources that require explicit cleanup beyond what GC provides. This is safe for the built-in handlers: Kafka's `ScramSaslServer.dispose()` is a no-op, and the OAUTHBEARER handler's per-connection state is similarly GC-safe. For reauthentication (KIP-368), the previous handler was already disposed at SUCCESS time, so a fresh handler is created for the new exchange. From fa9f0f0569b2a63e9eaff43394dc997117567935 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:24:33 +0000 Subject: [PATCH 37/52] docs(proposal): note TokenValidator SPI as planned future work Acknowledge the asymmetry between SCRAM's pluggable credential store and OAUTHBEARER's hardcoded validator. A TokenValidator SPI is planned as future work to enable token introspection and custom claim validators. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 91627f32..591ec86b 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -513,7 +513,7 @@ OAUTHBEARER is configured via `OauthBearerMechanismConfig`: - **No TLS configuration for the JWKS endpoint.** Kafka's `OAuthBearerValidatorCallbackHandler` uses an internal HTTP client with no TLS configuration surface. There is no way to configure custom trust stores or client certificates for HTTPS communication with the JWKS endpoint. The JVM's default trust store is used. This limitation is inherited from Kafka's callback handler and shared with the existing OAUTHBEARER validation filter. - **No rate limiting.** The handler does not implement rate limiting or brute-force protection for failed authentication attempts. The existing OAUTHBEARER validation filter has Caffeine-based rate limiting with exponential backoff that could serve as a reference for a future implementation. -- **Hardcoded `BrokerJwtValidator`.** The handler hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. +- **Hardcoded `BrokerJwtValidator`.** The handler hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. A `TokenValidator` SPI (analogous to the `ScramCredentialStore` SPI for SCRAM) would address this — the JWKS-based implementation would become the first-party provider, and the SPI would open the door to token introspection (RFC 7662) or custom claim validators. This is planned as future work. --- From 40deb2c3abfa8da4fc09e717c0cbbf32c00ab103 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:27:46 +0000 Subject: [PATCH 38/52] docs(proposal): reject __cluster_metadata credential store Kafka does not expose __cluster_metadata as a consumable topic, and DescribeUserScramCredentials deliberately does not return credential material. SCRAM credentials are write-only by design, so a broker-backed credential store is not possible. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 591ec86b..8d531520 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -871,6 +871,10 @@ A single `CredentialStore` interface serving both SCRAM and OAUTHBEARER was cons Instead, each mechanism family manages its own resources. The `MechanismHandlerFactory` is the point where mechanism-specific resources (credential stores, JWKS handlers) are injected. +### Credential store backed by Kafka's `__cluster_metadata` topic + +A `MetadataTopicScramCredentialStoreService` that consumes `UserScramCredentialRecord` from the `__cluster_metadata` topic was considered as a way to eliminate the credential island problem — the proxy could share the broker's own SCRAM credentials without separate provisioning. This was rejected because Kafka does not expose `__cluster_metadata` as a consumable topic, and the Admin API (`DescribeUserScramCredentials`) deliberately does not return the credential material (salt, serverKey, storedKey). This is by design: Kafka treats SCRAM credential material as write-only. There is no public API through which the proxy could obtain the credentials needed to perform SCRAM authentication. + ### Using @Plugin for mechanism handlers Making `MechanismHandlerFactory` a user-facing plugin (with `@Plugin` annotation and plugin discovery) was considered. This was rejected because: From ba1b432da807319f2b7d7a52dc567fa7e2ff4c01 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:36:49 +0000 Subject: [PATCH 39/52] docs(proposal): hash usernames for KeyStore aliases Use Base64URL-encoded SHA-256 hash of the username as the KeyStore alias instead of the raw username. JKS/PKCS12 are case-insensitive, keytool fails on special characters, non-ASCII is corrupted across tools, and Kafka does not apply SASLprep so any UTF-8 bytes can appear in usernames. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 8d531520..b928b66e 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -610,7 +610,7 @@ public class CredentialServiceTimeoutException extends CredentialLookupException #### Summary -The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry keyed by username. +The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry. The KeyStore alias for each entry is a Base64URL-encoded SHA-256 hash of the username, rather than the raw username. This avoids problems with KeyStore alias restrictions: JKS and PKCS12 are both case-insensitive (JKS lowercases aliases), `keytool` fails on aliases containing quotes, backslashes, commas, or periods, and non-ASCII characters are corrupted across tools. Since Kafka does not apply SASLprep, any UTF-8 bytes can appear in usernames, making raw aliases unreliable. The original username is stored within the JSON payload, so lookups hash the requested username and compare against the alias. Key features: From 75619be63fdc214fc209ef6b9faaa976380f0cb2 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:46:41 +0000 Subject: [PATCH 40/52] docs(proposal): use hex encoding for KeyStore alias hash Base64 is unsuitable because JKS/PKCS12 are case-insensitive. Use lowercase hex-encoded SHA-256 instead (64 chars, no case sensitivity issues). Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index b928b66e..2ac9d3ce 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -610,7 +610,7 @@ public class CredentialServiceTimeoutException extends CredentialLookupException #### Summary -The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry. The KeyStore alias for each entry is a Base64URL-encoded SHA-256 hash of the username, rather than the raw username. This avoids problems with KeyStore alias restrictions: JKS and PKCS12 are both case-insensitive (JKS lowercases aliases), `keytool` fails on aliases containing quotes, backslashes, commas, or periods, and non-ASCII characters are corrupted across tools. Since Kafka does not apply SASLprep, any UTF-8 bytes can appear in usernames, making raw aliases unreliable. The original username is stored within the JSON payload, so lookups hash the requested username and compare against the alias. +The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry. The KeyStore alias for each entry is a lowercase hex-encoded SHA-256 hash of the username, rather than the raw username. This avoids problems with KeyStore alias restrictions: JKS and PKCS12 are both case-insensitive (JKS lowercases aliases), `keytool` fails on aliases containing quotes, backslashes, commas, or periods, and non-ASCII characters are corrupted across tools. Since Kafka does not apply SASLprep, any UTF-8 bytes can appear in usernames, making raw aliases unreliable. The original username is stored within the JSON payload, so lookups hash the requested username and compare against the alias. Key features: From e4ab1aba80927808c612c513a3efdd55ff7bfae2 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:47:27 +0000 Subject: [PATCH 41/52] docs(proposal): add version field to credential JSON format Include a version field in the serialized JSON to allow the format to evolve while maintaining backwards compatibility. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 2ac9d3ce..27e693fa 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -610,7 +610,7 @@ public class CredentialServiceTimeoutException extends CredentialLookupException #### Summary -The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry. The KeyStore alias for each entry is a lowercase hex-encoded SHA-256 hash of the username, rather than the raw username. This avoids problems with KeyStore alias restrictions: JKS and PKCS12 are both case-insensitive (JKS lowercases aliases), `keytool` fails on aliases containing quotes, backslashes, commas, or periods, and non-ASCII characters are corrupted across tools. Since Kafka does not apply SASLprep, any UTF-8 bytes can appear in usernames, making raw aliases unreliable. The original username is stored within the JSON payload, so lookups hash the requested username and compare against the alias. +The first-party credential store provider, in the `kroxylicious-sasl-credential-store-provider-keystore` module, stores SCRAM credentials in a Java `KeyStore` file. It follows the project's established pattern of using KeyStores to store secrets. Each credential is serialized as JSON and stored as a `SecretKey` entry. The KeyStore alias for each entry is a lowercase hex-encoded SHA-256 hash of the username, rather than the raw username. This avoids problems with KeyStore alias restrictions: JKS and PKCS12 are both case-insensitive (JKS lowercases aliases), `keytool` fails on aliases containing quotes, backslashes, commas, or periods, and non-ASCII characters are corrupted across tools. Since Kafka does not apply SASLprep, any UTF-8 bytes can appear in usernames, making raw aliases unreliable. The original username is stored within the JSON payload, so lookups hash the requested username and compare against the alias. The JSON payload includes a `version` field to allow the format to be evolved while maintaining backwards compatibility. Key features: From 2f746a5fb038a5a4123651e6129e3d17ef37d075 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:52:21 +0000 Subject: [PATCH 42/52] docs(proposal): add trust boundary framing to security model Explicitly name the trust boundary shift: the proxy is now the authentication boundary and custodian of credential material. Recommend a comprehensive threat model as a separate activity. Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 27e693fa..211a2aed 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -752,6 +752,8 @@ The implementation is organized into three modules, following the same pattern a ## Security model +SASL termination fundamentally changes the proxy's trust level. Today the proxy does not hold authentication decision criteria — the broker holds SCRAM credentials, the IdP holds tokens, and the proxy passes bytes. With SASL termination, the proxy becomes the authentication boundary and the custodian of credential material: SCRAM credentials in memory and on disk, JWKS keys cached locally. This is a step change in what the proxy is responsible for protecting. The per-component mitigations below follow from this new role. A comprehensive threat model is recommended as a separate activity before production deployment. + ### Credential storage - **KeyStore encryption:** Credentials are stored in Java KeyStore files, encrypted with the KeyStore password. File-system permissions and KeyStore passwords are the primary access controls. From 16eafaf71225049d5d57d32285670c5fa250cd37 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 02:57:20 +0000 Subject: [PATCH 43/52] docs(proposal): note filter composition validation is out of scope Assisted-by: Claude Opus 4.6 (1M context) Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 211a2aed..faf08a9e 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -57,6 +57,8 @@ A key problem with any passthrough-based technique is that it depends on the ava This proposal aims to support the following SASL mechanisms: `SCRAM-SHA-256`, `SCRAM-SHA-512` and `OAUTHBEARER`. It also aims to be flexible, so as to allow other mechanisms to be supported in the future. +Validation of legal compositions of SASL-related filters within a filter chain is not in scope for this proposal. + The proposal is organized per-component. Each component section covers its summary, API surfaces, configuration, threats and mitigations, and known limitations. ### Component overview From 60911bb46895ab014cc947884b6bae0903aaaa64 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 15:36:03 +1200 Subject: [PATCH 44/52] Fix link; Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index faf08a9e..63205558 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -923,7 +923,7 @@ Supporting SASL GSSAPI was deferred because: [proposal-004]: 004-terminology-for-authentication.md [proposal-006]: 006-filter-api-to-expose-client-sasl-info.md -[proposal-072]: 070-routing-api.md +[proposal-072]: 072-routing-api.md [rfc4422]: https://www.rfc-editor.org/rfc/rfc4422 [rfc5802]: https://www.rfc-editor.org/rfc/rfc5802 [rfc5802-s6]: https://www.rfc-editor.org/rfc/rfc5802#section-6 From 7df521df472e383c6e0fed6fd99c55685ad50d40 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 30 Jul 2026 22:35:44 +0000 Subject: [PATCH 45/52] docs(proposal): internalize mechanism handling, add unsupported mechanism to state machine, restructure filter description Make mechanism handling internal to the filter rather than describing it as an SPI. Remove Component 2 (MechanismHandler extension point) and all references to internal types (MechanismHandler, MechanismHandlerFactory, MechanismConfig). Renumber components 3-7 to 2-6. Add unsupported mechanism handling to the state machine table and diagram: RequiringHandshake stays in RequiringHandshake, responds with UNSUPPORTED_SASL_MECHANISM and the supported mechanism list. Restructure Component 1 around request types (API_VERSIONS, SASL_HANDSHAKE, SASL_AUTHENTICATE, other) so the filter behaviour follows the client request sequence rather than listing features in isolation. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 311 +++++++----------------------- 1 file changed, 71 insertions(+), 240 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 63205558..9fa88916 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -63,21 +63,21 @@ The proposal is organized per-component. Each component section covers its summa ### Component overview -The `SaslTerminationFilter` (Component 1) intercepts SASL requests and manages per-connection authentication state via a sealed state machine. It delegates the actual authentication exchange to mechanism-specific `MechanismHandler` instances (Component 2), created per-connection by `MechanismHandlerFactory` implementations discovered via ServiceLoader. The OAUTHBEARER handler factory (Component 4) validates JWT tokens against a JWKS endpoint. The SCRAM handler factories (Component 3) use a `ScramCredentialStore` (Component 5) to look up stored credentials — a public SPI with a first-party KeyStore-backed provider (Component 6). +The `SaslTermination` filter (Component 1) intercepts SASL requests and manages per-connection authentication state via a sealed state machine. It handles each supported mechanism internally — SCRAM-SHA-256 and SCRAM-SHA-512 (Component 2) by delegating to Kafka's SASL framework with credentials from a `ScramCredentialStore` (Component 4), and OAUTHBEARER (Component 3) by validating JWT tokens against a JWKS endpoint. The `ScramCredentialStore` is a public SPI with a first-party KeyStore-backed provider (Component 5). Mechanism handling is internal to the filter and is not a public API. The implementation spans three modules. Their dependencies: ```mermaid graph LR - ST["kroxylicious-sasl-termination
(Components 1–4)"] - CS["kroxylicious-sasl-credential-store
(Component 5)"] - KP["kroxylicious-sasl-credential-store-provider-keystore
(Component 6)"] + ST["kroxylicious-sasl-termination
(Components 1–3)"] + CS["kroxylicious-sasl-credential-store
(Component 4)"] + KP["kroxylicious-sasl-credential-store-provider-keystore
(Component 5)"] ST --> CS KP --> CS ``` -The key types and their relationships across these modules: +The key public types and their relationships across these modules: ```mermaid classDiagram @@ -87,30 +87,6 @@ classDiagram class SaslTerminationFilter { <> } - class State { - <> - RequiringHandshake - RequiringAuthenticate - Authenticated - Failed - } - class MechanismHandlerFactory { - <> - +mechanismName() String - +initialize(MechanismConfig) - +createHandler() MechanismHandler - } - class MechanismHandler { - <> - +handleAuthenticate(byte[]) CompletionStage~AuthenticationResult~ - } - class AuthenticationResult { - <> - CHALLENGE / SUCCESS / FAILURE - } - class MechanismConfig { - <> - } } namespace CredentialStoreSPI { @@ -134,12 +110,7 @@ classDiagram } } - SaslTerminationFilter *-- State - SaslTerminationFilter ..> MechanismHandlerFactory : discovers via ServiceLoader - MechanismHandlerFactory --> MechanismHandler : creates per connection - MechanismHandler --> AuthenticationResult : returns - MechanismHandlerFactory ..> MechanismConfig : configured by - MechanismHandler ..> ScramCredentialStore : looks up credentials (SCRAM) + SaslTerminationFilter ..> ScramCredentialStore : looks up SCRAM credentials ScramCredentialStoreService --> ScramCredentialStore : builds ScramCredentialStore --> ScramCredential : returns @@ -150,16 +121,23 @@ classDiagram #### Summary -The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts all requests on a connection. For `SASL_HANDSHAKE` and `SASL_AUTHENTICATE` requests, it authenticates clients at the proxy, short-circuiting those exchanges without forwarding them to the broker. For all other request types, the filter enforces the security barrier: if the client has not completed authentication, the request is rejected and the connection is closed; if the session lifetime has elapsed, the filter does the same, requiring the client to reauthenticate. In practice the unauthenticated rejection should never be reached for non-SASL requests, because failed authentication already closes the connection — but the check exists for defence in depth. Multiple mechanisms are configured within a single filter instance because the Kafka SASL protocol requires it: the client sends a `SaslHandshakeRequest` naming its chosen mechanism, and the server responds with the set of supported mechanisms. A filter-per-mechanism model would not work because no single filter would have the complete set of supported mechanisms to advertise in the `SaslHandshakeResponse`. +The `SaslTermination` filter is a `@Plugin`-annotated `FilterFactory` that intercepts all requests on a connection. It authenticates clients at the proxy, short-circuiting SASL exchanges without forwarding them to the broker. The filter itself is mechanism-agnostic; but mechanism-specific logic is internal to the filter module and is not a public API. Multiple mechanisms are configured within a single filter instance because the Kafka SASL protocol requires it: the client sends a `SaslHandshakeRequest` naming its chosen mechanism, and if that mechanism is not supported the server responds with the set of supported mechanisms. -Key features: +Per-connection authentication state is modelled as a sealed state machine (see [State machine](#state-machine)). The filter also supports [KIP-368][kip368] reauthentication (see [Reauthentication](#reauthentication-kip-368)). + +#### Request handling + +A Kafka client connecting via SASL follows a defined request sequence. The filter handles each request type as follows: -- **Security barrier.** Until a client has successfully authenticated, the only requests permitted are `API_VERSIONS`, `SASL_HANDSHAKE`, and `SASL_AUTHENTICATE`. All other request types are rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. -- **Fail closed on unknown versions.** If a `SASL_HANDSHAKE` or `SASL_AUTHENTICATE` request arrives with an API version outside the range known to the filter, the filter rejects the request and closes the connection. This prevents a future protocol version from bypassing the filter's security logic. -- **State machine.** Per-connection authentication state is modelled as a sealed interface with four concrete states, preventing invalid transitions at compile time. -- **Reauthentication (KIP-368).** When `maxTimeBeforeReauth` is configured, the filter includes a `sessionLifetimeMs` value in `SaslAuthenticateResponse` (v1+), informing the client when to reauthenticate. Sessions that expire without reauthentication are rejected and closed. -- **Mechanism dispatch.** The filter delegates each authentication exchange to a `MechanismHandler` obtained from the appropriate `MechanismHandlerFactory` (see Component 2). The filter itself is mechanism-agnostic. -- **Steady-state optimization.** Once a client is in the `Authenticated` state with no session expiry configured, the filter uses `shouldHandleRequest` to avoid deserializing subsequent requests, letting them pass through without filter overhead. +**`API_VERSIONS`** — Always accepted, regardless of authentication state. The filter modifies the response to remove APIs that are not meaningful when SASL is terminated at the proxy: the delegation token APIs (`CreateDelegationToken`, `RenewDelegationToken`, `ExpireDelegationToken`, `DescribeDelegationToken`) and `AlterUserScramCredentials` (since the credential store is not writable via the Kafka protocol). `DescribeUserScramCredentials` _can_ be answered by the filter from its own credential store. + +**`SASL_HANDSHAKE`** — Accepted in the initial (unauthenticated) state and in the authenticated state (for reauthentication). The client names its chosen mechanism. If the mechanism is supported, the filter transitions to the authenticating state. If the mechanism is not supported, the filter responds with `UNSUPPORTED_SASL_MECHANISM` and the list of supported mechanisms; the state does not change and the client may retry with a different mechanism. If the request arrives with an API version outside the range known to the filter, the request is rejected and the connection is closed — this prevents a future protocol version from bypassing the filter's security logic. + +**`SASL_AUTHENTICATE`** — Accepted only in the authenticating state. The filter delegates to internal per-mechanism logic. For multi-round mechanisms (e.g. SCRAM), the exchange loops with challenge responses until it completes. On success, the filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters. On failure, the connection is closed. As with `SASL_HANDSHAKE`, unknown API versions are rejected and the connection is closed. + +**All other requests** — Accepted only after authentication succeeds. If the client has not authenticated, the request is rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. If the session has expired (see [Reauthentication](#reauthentication-kip-368)), the same happens. In practice, the pre-authentication rejection should never be reached for non-SASL requests because failed authentication already closes the connection; the check exists for defence in depth. + +**Steady-state optimization:** Once a client is authenticated with no session expiry configured, the filter avoids deserializing subsequent requests, letting them pass through without filter overhead. #### State machine @@ -168,9 +146,10 @@ The filter maintains per-connection state using a sealed interface `State` with | From state | Triggering event | To state | |------------|------------------|----------| | **RequiringHandshake** | `SASL_HANDSHAKE` with supported mechanism | **RequiringAuthenticate** | -| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `CHALLENGE` | **RequiringAuthenticate** (loop) | -| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `SUCCESS` | **Authenticated** | -| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → handler returns `FAILURE` | **Failed** | +| **RequiringHandshake** | `SASL_HANDSHAKE` with unsupported mechanism | **RequiringHandshake** (no change) | +| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → outcome is `CHALLENGE` | **RequiringAuthenticate** (loop) | +| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → outcome is `SUCCESS` | **Authenticated** | +| **RequiringAuthenticate** | `SASL_AUTHENTICATE` → outcome is `FAILURE` | **Failed** | | **Authenticated** | `SASL_HANDSHAKE` (reauthentication) | **RequiringAuthenticate** | | **Authenticated** | non-SASL request, session not expired | forward to broker | | **Authenticated** | non-SASL request, session expired | reject and close | @@ -180,6 +159,7 @@ The filter maintains per-connection state using a sealed interface `State` with stateDiagram-v2 direction LR [*] --> RequiringHandshake + RequiringHandshake --> RequiringHandshake : unsupported mechanism RequiringHandshake --> RequiringAuthenticate : supported mechanism note right of RequiringAuthenticate : Loops on CHALLENGE RequiringAuthenticate --> Authenticated : SUCCESS @@ -193,8 +173,8 @@ stateDiagram-v2 **In-flight requests at expiry:** The filter checks expiry before forwarding, so the request that triggers the expiry check never reaches the broker. Previously-forwarded requests whose responses are still in flight will be delivered to the client before the connection closes. -- **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests, which negotiate the mechanism and transition to `RequiringAuthenticate`. -- **RequiringAuthenticate:** Accepts `SASL_AUTHENTICATE` requests. Loops back to itself for multi-round mechanisms (e.g. SCRAM). Carries a reference to the `MechanismHandler` for the negotiated mechanism. +- **RequiringHandshake:** Initial state. Accepts `SASL_HANDSHAKE` requests. If the mechanism is supported, transitions to `RequiringAuthenticate`. If the mechanism is unsupported, responds with `UNSUPPORTED_SASL_MECHANISM` and the list of supported mechanisms; stays in `RequiringHandshake` so the client can retry. +- **RequiringAuthenticate:** Accepts `SASL_AUTHENTICATE` requests. Loops back to itself for multi-round mechanisms (e.g. SCRAM). - **Authenticated:** Success state. The filter calls `filterContext.clientSaslAuthenticationSuccess(mechanism, subject)` to propagate the authenticated identity to downstream filters, then forwards all subsequent requests. If reauthentication is configured, this state also stores the session expiry time and allows transition back to `RequiringAuthenticate` via a new `SASL_HANDSHAKE`. - **Failed:** Terminal failure state. The connection is closed. @@ -204,13 +184,13 @@ The filter supports [KIP-368][kip368] reauthentication. **Session lifetime computation:** The effective session lifetime is the minimum of: 1. The configured `maxTimeBeforeReauth` value. -2. The handler-reported credential/token lifetime (e.g. the JWT token's expiry for OAUTHBEARER). +2. The mechanism-reported credential/token lifetime (e.g. the JWT token's expiry for OAUTHBEARER). If either value is zero (no opinion / no expiry), the other is used. If both are zero, no reauthentication is required. Reauthentication is a protocol-level feature, not mechanism-specific — all mechanisms support it. The difference is the session lifetime source: -- **SCRAM:** Credentials do not expire, so the handler reports no lifetime. `maxTimeBeforeReauth` is the sole source of session lifetime. Without it configured, SCRAM sessions never require reauthentication. -- **OAUTHBEARER:** Tokens have an inherent expiry. The handler reports the token's remaining lifetime, and the effective session lifetime is `min(maxTimeBeforeReauth, tokenExpiry)`. Even without `maxTimeBeforeReauth`, sessions expire when the token does. +- **SCRAM:** Credentials do not expire, so no lifetime is reported. `maxTimeBeforeReauth` is the sole source of session lifetime. Without it configured, SCRAM sessions never require reauthentication. +- **OAUTHBEARER:** Tokens have an inherent expiry. The token's remaining lifetime is reported, and the effective session lifetime is `min(maxTimeBeforeReauth, tokenExpiry)`. Even without `maxTimeBeforeReauth`, sessions expire when the token does. **Client behaviour:** Standard Kafka clients (4.0+) handle reauthentication transparently via the `Selector`. When the session nears expiry, the client sends a new `SASL_HANDSHAKE` + `SASL_AUTHENTICATE` sequence over the existing connection. This is invisible to application code. @@ -223,7 +203,6 @@ The filter is a standard Kroxylicious `FilterFactory` plugin. It does not define - `FilterFactory` (from `kroxylicious-api`) -- the standard filter factory contract. - `RequestFilter` (from `kroxylicious-api`) -- for intercepting requests. - `FilterContext.clientSaslAuthenticationSuccess()` / `clientSaslAuthenticationFailure()` (from `kroxylicious-api`, added by Proposal 006) -- to propagate authentication outcomes. -- `MechanismHandlerFactory` (internal, see Component 2) -- for mechanism dispatch. #### Configuration @@ -231,12 +210,12 @@ The filter is configured via `SaslTerminationConfig`: | Option | Type | Required | Default | Description | |--------|------|----------|---------|-------------| -| `mechanisms` | `List` | Yes | -- | List of mechanism configurations. Each entry includes a `mechanism` field (the IANA-registered mechanism name) and mechanism-specific configuration. At least one entry is required. | +| `mechanisms` | `List` | Yes | -- | List of mechanism configurations. Each entry includes a `mechanism` field (the IANA-registered mechanism name, e.g. `SCRAM-SHA-256`, `OAUTHBEARER`) and mechanism-specific configuration. At least one entry is required. | | `maxTimeBeforeReauth` | `Duration` | No | disabled | Maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. | | `fixedAuthDelay` | `Duration` | No | `200ms` | Fixed delay applied to all authentication rounds to prevent timing side-channel attacks that could enable user enumeration. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. | | `subjectBuilder` | `SaslSubjectBuilderService` | No | `DEFAULT_SUBJECT_BUILDER` | Plugin for constructing the `Subject` from authentication results. Defaults to `DEFAULT_SUBJECT_BUILDER`, consistent with the existing SASL inspection filter. | -The `mechanisms` list elements are polymorphic. Jackson name-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism")`) resolves the concrete type from the `mechanism` field, which is the IANA-registered mechanism name (e.g. `SCRAM-SHA-256`, `OAUTHBEARER`). This also selects which `MechanismHandlerFactory` handles the exchange. +The `mechanisms` list elements are polymorphic. Jackson name-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism")`) resolves the concrete type from the `mechanism` field, which is the IANA-registered mechanism name (e.g. `SCRAM-SHA-256`, `OAUTHBEARER`). **Example configuration:** @@ -274,144 +253,16 @@ filters: --- -### Component 2: MechanismHandler internal extension point - -#### Summary - -The filter delegates the actual authentication exchange to mechanism-specific handlers, discovered via an internal extension point. This extension point provides internal extensibility for adding new mechanism support without modifying the filter itself. - -These are **not** intended to be configurable by end uses (no `@Plugin` annotation). The intention behind this decision is to encourage a small number of secure, high-quality implementations, one for each mechanism. Allowing pluggable implementations would make auditing for correctness and security significantly harder. - -#### API surfaces - -The extension point consists of three types, all in the `io.kroxylicious.filter.sasl.termination.mechanism` package within the `kroxylicious-sasl-termination` module. - -**`MechanismHandler`** -- handles the authentication exchange for a single connection. Instances are per-connection and not thread-safe. - -```java -public interface MechanismHandler { - - String mechanismName(); - - CompletionStage handleAuthenticate(byte[] authBytes); - - void dispose(); -} -``` - -**`MechanismHandler` lifecycle:** The filter calls `dispose()` on the handler after SUCCESS (the handler is no longer needed once the client is authenticated) and after FAILURE (the connection is about to close). It is *not* called on raw connection close (e.g. client disconnects mid-exchange) because the `Filter` API has no connection-close hook — the handler becomes unreachable and is garbage collected. Handler implementations must therefore not hold resources that require explicit cleanup beyond what GC provides. This is safe for the built-in handlers: Kafka's `ScramSaslServer.dispose()` is a no-op, and the OAUTHBEARER handler's per-connection state is similarly GC-safe. - -For reauthentication (KIP-368), the previous handler was already disposed at SUCCESS time, so a fresh handler is created for the new exchange. - -**`MechanismHandlerFactory`** -- manages mechanism-specific resources and creates handler instances. Discovered via `ServiceLoader`. - -```java -public interface MechanismHandlerFactory extends AutoCloseable { - - String mechanismName(); - - void initialize(MechanismConfig config, FilterFactoryContext context, Clock clock) - throws PluginConfigurationException; - - MechanismHandler createHandler(); - - @Override - void close(); -} -``` - -Each factory: -1. Reports its IANA-registered mechanism name via `mechanismName()`. -2. Receives mechanism-specific configuration at `initialize()` time and creates whatever resources the mechanism requires (credential stores, JWKS callback handlers, etc.). -3. Creates per-connection `MechanismHandler` instances via `createHandler()`, injecting shared resources. -4. Releases resources on `close()`. - -**`AuthenticationResult`** -- the outcome of processing a single SASL authenticate request. - -```java -public record AuthenticationResult( - Outcome outcome, - byte[] responseBytes, - @Nullable String authorizationId, - @Nullable String errorMessage, - long sessionLifetimeMs) { - - public enum Outcome { CHALLENGE, SUCCESS, FAILURE } - - public static AuthenticationResult challenge(byte[] responseBytes); - public static AuthenticationResult success(byte[] responseBytes, String authorizationId); - public static AuthenticationResult success(byte[] responseBytes, String authorizationId, - long sessionLifetimeMs); - public static AuthenticationResult failure(byte[] responseBytes, String errorMessage); -} -``` - -**`MechanismConfig`** -- sealed interface for mechanism-specific configuration, using Jackson name-based polymorphism on the `mechanism` field: - -```java -@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism") -@JsonSubTypes({ - @JsonSubTypes.Type(value = ScramSha256MechanismConfig.class, name = "SCRAM-SHA-256"), - @JsonSubTypes.Type(value = ScramSha512MechanismConfig.class, name = "SCRAM-SHA-512"), - @JsonSubTypes.Type(value = OauthBearerMechanismConfig.class, name = "OAUTHBEARER") -}) -public sealed interface MechanismConfig - permits ScramMechanismConfig, OauthBearerMechanismConfig { -} -``` - -`ScramMechanismConfig` is an abstract base class whose constructor accepts the mechanism name. The per-variant subclasses contain only a default constructor: - -```java -public abstract sealed class ScramMechanismConfig implements MechanismConfig - permits ScramSha256MechanismConfig, ScramSha512MechanismConfig { - - private final String mechanism; - - protected ScramMechanismConfig(String mechanism) { - this.mechanism = mechanism; - } - - // credentialStore, credentialStoreConfig fields... -} - -public final class ScramSha256MechanismConfig extends ScramMechanismConfig { - public ScramSha256MechanismConfig() { super("SCRAM-SHA-256"); } -} - -public final class ScramSha512MechanismConfig extends ScramMechanismConfig { - public ScramSha512MechanismConfig() { super("SCRAM-SHA-512"); } -} -``` - -#### ServiceLoader discovery - -Factories are registered in `META-INF/services/io.kroxylicious.filter.sasl.termination.mechanism.MechanismHandlerFactory`. At filter factory initialization time, the `SaslTermination` filter factory loads all registered factories, matches them to the mechanism names present in the user's configuration, and calls `initialize()` on each matched factory. - -#### Built-in mechanism handlers - -| Mechanism | Factory | Handler | Specification | -|-----------|---------|---------|---------------| -| `SCRAM-SHA-256` | `ScramSha256HandlerFactory` | `ScramHandler` | [RFC 5802][rfc5802] | -| `SCRAM-SHA-512` | `ScramSha512HandlerFactory` | `ScramHandler` | [RFC 5802][rfc5802] | -| `OAUTHBEARER` | `OauthBearerHandlerFactory` | `OauthBearerHandler` | [RFC 6750][rfc6750] / [RFC 7628][rfc7628] | - -#### Known limitations - -- Adding a new mechanism requires adding a new `MechanismHandlerFactory` implementation within the `kroxylicious-sasl-termination` module, a new `MechanismConfig` subtype, and updating the sealed permit list. This is intentional. - ---- - -### Component 3: SCRAM mechanism handler +### Component 2: SCRAM mechanism support #### Summary -The SCRAM mechanism handler (`ScramHandler`) implements multi-round `SCRAM-SHA-256` and `SCRAM-SHA-512` authentication by delegating to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework. Two factories -- `ScramSha256HandlerFactory` and `ScramSha512HandlerFactory` -- manage the credential store lifecycle and create per-connection handler instances. +The filter implements multi-round `SCRAM-SHA-256` and `SCRAM-SHA-512` authentication by delegating to Apache Kafka's own `SaslServer` implementation via the JSSE/SASL framework. Key features: -- **Multi-round SCRAM exchange.** SCRAM is a challenge-response protocol. The handler processes the client-first-message (round 1) and subsequent rounds, returning `CHALLENGE` until the exchange completes. -- **Delegation to Kafka's `SaslServer`.** The handler does not reimplement SCRAM. It creates a Kafka `SaslServer` with a `CallbackHandler` that supplies the looked-up credential, then processes all messages through it. This benefits from Kafka's battle-tested implementation. +- **Multi-round SCRAM exchange.** SCRAM is a challenge-response protocol. The filter processes the client-first-message (round 1) and subsequent rounds, returning challenge responses until the exchange completes. +- **Delegation to Kafka's `SaslServer`.** The filter does not reimplement SCRAM. It creates a Kafka `SaslServer` with a `CallbackHandler` that supplies the looked-up credential, then processes all messages through it. This benefits from Kafka's battle-tested implementation. - **Timing side-channel mitigation.** A configurable fixed delay (`fixedAuthDelay`) is applied to all authentication rounds to prevent attackers from distinguishing existing from non-existing users by measuring response times. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. #### Authentication flow @@ -421,28 +272,13 @@ Key features: #### API surfaces -The SCRAM handler factories use: +SCRAM mechanism support uses: -- `MechanismHandlerFactory` / `MechanismHandler` (internal, Component 2) -- the internal extension point. -- `ScramCredentialStore` (public SPI, Component 5) -- for credential lookup. The factory resolves the credential store plugin at `initialize()` time using the Kroxylicious plugin system (`@PluginImplName` / `@PluginImplConfig`). +- `ScramCredentialStore` (public SPI, Component 4) -- for credential lookup. The credential store plugin is resolved at initialization time using the Kroxylicious plugin system (`@PluginImplName` / `@PluginImplConfig`). #### Configuration -SCRAM mechanisms are configured via `ScramMechanismConfig` (see Component 2 for the `ScramSha256MechanismConfig` / `ScramSha512MechanismConfig` subclasses). The base class carries the credential store configuration: - -```java -public abstract sealed class ScramMechanismConfig implements MechanismConfig - permits ScramSha256MechanismConfig, ScramSha512MechanismConfig { - - @JsonProperty(required = true) - @PluginImplName(ScramCredentialStoreService.class) - private String credentialStore; - - @JsonProperty(required = true) - @PluginImplConfig(implNameProperty = "credentialStore") - private Object credentialStoreConfig; -} -``` +SCRAM mechanisms are configured within the filter's `mechanisms` list. The SCRAM-specific configuration carries the credential store reference: | Option | Type | Required | Default | Description | |--------|------|----------|---------|-------------| @@ -453,9 +289,9 @@ public abstract sealed class ScramMechanismConfig implements MechanismConfig | Threat | Mitigation | |--------|------------| -| Username enumeration -- an attacker distinguishes existing from non-existing users by observing different error messages. | When a user is not found, the handler returns a generic `"Authentication failed"` error message identical to the message returned for incorrect credentials. | +| Username enumeration -- an attacker distinguishes existing from non-existing users by observing different error messages. | When a user is not found, the filter returns a generic `"Authentication failed"` error message identical to the message returned for incorrect credentials. | | Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter applies a configurable fixed delay (`fixedAuthDelay`) to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. | -| SCRAM protocol correctness -- a bug in the SCRAM implementation could allow authentication bypass or credential leakage. | Delegated to Kafka's own `SaslServer`, which is widely deployed and well-tested. The handler is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. | +| SCRAM protocol correctness -- a bug in the SCRAM implementation could allow authentication bypass or credential leakage. | Delegated to Kafka's own `SaslServer`, which is widely deployed and well-tested. The filter is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. | #### Known limitations @@ -463,33 +299,28 @@ public abstract sealed class ScramMechanismConfig implements MechanismConfig --- -### Component 4: OAUTHBEARER mechanism handler +### Component 3: OAUTHBEARER mechanism support #### Summary -The OAUTHBEARER mechanism handler (`OauthBearerHandler`) validates JWT bearer tokens at the proxy without forwarding them to the broker. The `OauthBearerHandlerFactory` manages the JWKS endpoint configuration and callback handler lifecycle. +The filter validates JWT bearer tokens at the proxy without forwarding them to the broker. Key features: -- **JWT validation via Kafka's `OAuthBearerValidatorCallbackHandler`.** The factory configures the callback handler at `initialize()` time with the JWKS endpoint, expected audience/issuer, and refresh settings. Per-connection handlers receive the shared callback handler and use it to create a `SaslServer` via the JSSE/SASL framework. -- **Token lifetime extraction for reauthentication.** After successful authentication, the handler extracts the token's remaining lifetime from the `SaslServer`'s negotiated `CREDENTIAL.LIFETIME.MS` property, returning it via `AuthenticationResult.sessionLifetimeMs` for use in session lifetime computation (see [Reauthentication](#reauthentication-kip-368)). -- **No credential store required.** OAUTHBEARER is architecturally simpler than SCRAM -- the factory's only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). +- **JWT validation via Kafka's `OAuthBearerValidatorCallbackHandler`.** The filter configures the callback handler at initialization time with the JWKS endpoint, expected audience/issuer, and refresh settings. Per-connection authentication uses the shared callback handler to create a `SaslServer` via the JSSE/SASL framework. +- **Token lifetime extraction for reauthentication.** After successful authentication, the token's remaining lifetime is extracted from the `SaslServer`'s negotiated `CREDENTIAL.LIFETIME.MS` property for use in session lifetime computation (see [Reauthentication](#reauthentication-kip-368)). +- **No credential store required.** OAUTHBEARER is architecturally simpler than SCRAM — the only external dependency is the JWKS endpoint, and authentication is typically single-round (client sends token, server validates it). **Key differences from the existing OAUTHBEARER validation filter:** -- The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, the termination handler validates tokens and _short-circuits_ -- the broker never sees a SASL exchange. -- The handler factory owns its callback handler and JWKS configuration, receiving them at `initialize()` time rather than requiring a credential store. +- The existing validation filter validates tokens then _forwards_ the SASL exchange to the broker. It is fundamentally a SASL passthrough technique. In contrast, SASL termination validates tokens and _short-circuits_ — the broker never sees a SASL exchange. #### API surfaces -The OAUTHBEARER handler factory uses: - -- `MechanismHandlerFactory` / `MechanismHandler` (internal, Component 2) -- the internal extension point. - -It does not use the `ScramCredentialStore` SPI. Token validation is performed entirely by Kafka's `OAuthBearerValidatorCallbackHandler`. +OAUTHBEARER mechanism support does not use the `ScramCredentialStore` SPI. Token validation is performed entirely by Kafka's `OAuthBearerValidatorCallbackHandler`. #### Configuration -OAUTHBEARER is configured via `OauthBearerMechanismConfig`: +The OAUTHBEARER mechanism is configured within the filter's `mechanisms` list with the following options: | Option | Type | Required | Default | Description | |--------|------|----------|---------|-------------| @@ -508,18 +339,18 @@ OAUTHBEARER is configured via `OauthBearerMechanismConfig`: | Threat | Mitigation | |--------|------------| -| Token from wrong audience or issuer -- a JWT issued for a different service or identity provider is presented to the proxy. | Both `expectedAudience` and `expectedIssuer` are required fields. The handler rejects tokens that do not match. | +| Token from wrong audience or issuer -- a JWT issued for a different service or identity provider is presented to the proxy. | Both `expectedAudience` and `expectedIssuer` are required fields. Tokens that do not match are rejected. | | JWKS endpoint compromise -- an attacker controls the JWKS endpoint and supplies signing keys for forged tokens. | Mitigated operationally: the JWKS endpoint URL is set by the proxy administrator, not by clients. TLS protects the endpoint in transit (using the JVM's default trust store). | #### Known limitations - **No TLS configuration for the JWKS endpoint.** Kafka's `OAuthBearerValidatorCallbackHandler` uses an internal HTTP client with no TLS configuration surface. There is no way to configure custom trust stores or client certificates for HTTPS communication with the JWKS endpoint. The JVM's default trust store is used. This limitation is inherited from Kafka's callback handler and shared with the existing OAUTHBEARER validation filter. -- **No rate limiting.** The handler does not implement rate limiting or brute-force protection for failed authentication attempts. The existing OAUTHBEARER validation filter has Caffeine-based rate limiting with exponential backoff that could serve as a reference for a future implementation. -- **Hardcoded `BrokerJwtValidator`.** The handler hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. A `TokenValidator` SPI (analogous to the `ScramCredentialStore` SPI for SCRAM) would address this — the JWKS-based implementation would become the first-party provider, and the SPI would open the door to token introspection (RFC 7662) or custom claim validators. This is planned as future work. +- **No rate limiting.** The filter does not implement rate limiting or brute-force protection for failed authentication attempts. The existing OAUTHBEARER validation filter has Caffeine-based rate limiting with exponential backoff that could serve as a reference for a future implementation. +- **Hardcoded `BrokerJwtValidator`.** The implementation hardcodes `BrokerJwtValidator` as the JWT validator. The existing OAUTHBEARER validation filter allows this to be overridden via `jwtValidatorClass` for custom claim validation logic. A `TokenValidator` SPI (analogous to the `ScramCredentialStore` SPI for SCRAM) would address this — the JWKS-based implementation would become the first-party provider, and the SPI would open the door to token introspection (RFC 7662) or custom claim validators. This is planned as future work. --- -### Component 5: ScramCredentialStore SPI (public plugin API) +### Component 4: ScramCredentialStore SPI (public plugin API) #### Summary @@ -608,7 +439,7 @@ public class CredentialServiceTimeoutException extends CredentialLookupException --- -### Component 6: KeyStore credential store provider +### Component 5: KeyStore credential store provider #### Summary @@ -623,7 +454,7 @@ Key features: #### API surfaces -- Implements `ScramCredentialStoreService` (from Component 5). +- Implements `ScramCredentialStoreService` (from Component 4). - Annotated with `@Plugin` for discovery by the Kroxylicious plugin system. #### Configuration @@ -742,15 +573,15 @@ Note: `ScramCredential` uses defensive copies for `byte[]` fields and redacts `t --- -### Component 7: Module architecture +### Component 6: Module architecture The implementation is organized into three modules, following the same pattern as the existing KMS modules (`kroxylicious-kms`, `kroxylicious-kms-provider-*`): | Module | Contents | Components | |--------|----------|------------| -| `kroxylicious-filters/kroxylicious-sasl-termination` | Filter, state machine, `MechanismHandler` / `MechanismHandlerFactory` internal SPI, `MechanismConfig` sealed hierarchy, and all built-in mechanism handler implementations (SCRAM, OAUTHBEARER). | 1, 2, 3, 4 | -| `kroxylicious-sasl-credential-store` | Public API: `ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, exception hierarchy. No implementation, no Kafka dependencies. | 5 | -| `kroxylicious-sasl-credential-store-providers/kroxylicious-sasl-credential-store-provider-keystore` | First-party SCRAM credential provider: Java KeyStore-backed `ScramCredentialStoreService` implementation with `KeystoreCredentialTool` CLI. | 6 | +| `kroxylicious-filters/kroxylicious-sasl-termination` | Filter, state machine, mechanism-specific configuration, and all built-in mechanism support (SCRAM, OAUTHBEARER). | 1, 2, 3 | +| `kroxylicious-sasl-credential-store` | Public API: `ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, exception hierarchy. No implementation, no Kafka dependencies. | 4 | +| `kroxylicious-sasl-credential-store-providers/kroxylicious-sasl-credential-store-provider-keystore` | First-party SCRAM credential provider: Java KeyStore-backed `ScramCredentialStoreService` implementation with `KeystoreCredentialTool` CLI. | 5 | ## Security model @@ -761,7 +592,7 @@ SASL termination fundamentally changes the proxy's trust level. Today the proxy - **KeyStore encryption:** Credentials are stored in Java KeyStore files, encrypted with the KeyStore password. File-system permissions and KeyStore passwords are the primary access controls. - **PasswordProvider abstraction:** Production deployments should use file-based passwords rather than inline passwords in configuration. The `PasswordProvider` interface supports both. - **File permission enforcement:** On POSIX systems, the credential store checks the KeyStore file's permissions before loading it. By default, group or world read/write permissions are rejected (`0600` or stricter required). This prevents accidental exposure of credential material through overly permissive file modes. The required permission level is configurable via the `KROXYLICIOUS_DANGEROUSLY_CHANGE_PERMISSION_CHECK` environment variable, which can be set to `0640` to allow group-readable files. This is necessary on OpenShift, where the `restricted-v2` SCC runs containers as an arbitrary UID while Secret volume files are owned by root — requiring group-readable permissions (`defaultMode: 0440` with `fsGroup`) for the container process to access them. The environment variable is set in the PodSpec by the `kroxylicious-operator`, keeping the trust chain secure: operator → pod spec → env var → policy, with no writable config file in the loop. Using a config file for this setting would create a bootstrapping problem — if the config file itself were group-writable, an attacker could downgrade the permission policy. -- **In-memory handling:** `ScramCredential` uses defensive copies for `byte[]` fields (correctness measure against accidental mutation) and `toString()` redacts sensitive fields (prevents log leakage). Credential material in the JVM heap is an accepted risk — see Component 6 threat discussion. +- **In-memory handling:** `ScramCredential` uses defensive copies for `byte[]` fields (correctness measure against accidental mutation) and `toString()` redacts sensitive fields (prevents log leakage). Credential material in the JVM heap is an accepted risk — see Component 5 threat discussion. ### SCRAM protocol correctness @@ -845,11 +676,11 @@ This implementation uses several Kafka APIs that are not part of the [published **`org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServerProvider`** — Called once (`initialize()`) to register the OAUTHBEARER SASL mechanism with the JVM's security provider infrastructure. The existing OAUTHBEARER validation filter uses this in the same way. There is no public API alternative. [Proposal 116][proposal-116] would copy this into the Kroxylicious namespace, giving stability control, but the functional dependency on Kafka's JSSE provider registration code remains. -**`org.apache.kafka.common.security.scram.internals.ScramMechanism`** — An enum identifying SCRAM-SHA-256 and SCRAM-SHA-512. Used internally by the SCRAM handler factories and the keystore credential manager. There is no public API equivalent. [Proposal 116][proposal-116] would own this type, but it is a trivial enum that could equally be replaced with a Kroxylicious-native type. +**`org.apache.kafka.common.security.scram.internals.ScramMechanism`** — An enum identifying SCRAM-SHA-256 and SCRAM-SHA-512. Used internally by the SCRAM mechanism support and the keystore credential manager. There is no public API equivalent. [Proposal 116][proposal-116] would own this type, but it is a trivial enum that could equally be replaced with a Kroxylicious-native type. **`org.apache.kafka.common.security.scram.internals.ScramFormatter`** — Used by `KeystoreCredentialManager` to derive salted passwords, server keys, and stored keys from plaintext passwords. This is the only implementation of SCRAM key derivation available in the Kafka client library. There is no public API equivalent. [Proposal 116][proposal-116] would copy this into the Kroxylicious namespace, but unlike the protocol data classes, `ScramFormatter` is a functional security implementation (PBKDF2, HMAC) — the maintenance burden of keeping it current remains. -All of these dependencies are contained within the implementation modules. The public SPI types (`ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`, `MechanismHandler`, `MechanismHandlerFactory`, `AuthenticationResult`) do not reference any Kafka types. Implementors of the credential store SPI are not transitively exposed to Kafka internal APIs. +All of these dependencies are contained within the implementation modules. The public SPI types (`ScramCredentialStore`, `ScramCredentialStoreService`, `ScramCredential`) do not reference any Kafka types. Implementors of the credential store SPI are not transitively exposed to Kafka internal APIs. The `KeystoreCredentialManager` class does expose `ScramMechanism` in its public method signatures (`addUser`, `updatePassword`, `generateKeyStore`, `generateScramCredential`). This class is in the provider module, not the SPI, so it is not part of the formal public API contract — but external code that uses the credential manager directly would take a dependency on this internal Kafka type. @@ -873,18 +704,18 @@ A single `CredentialStore` interface serving both SCRAM and OAUTHBEARER was cons - OAUTHBEARER uses token validation against a JWKS endpoint (no stored credentials at all). - A generic interface would either be too abstract to be useful or would leak mechanism-specific concepts into the abstraction. -Instead, each mechanism family manages its own resources. The `MechanismHandlerFactory` is the point where mechanism-specific resources (credential stores, JWKS handlers) are injected. +Instead, each mechanism family manages its own resources internally within the filter. ### Credential store backed by Kafka's `__cluster_metadata` topic A `MetadataTopicScramCredentialStoreService` that consumes `UserScramCredentialRecord` from the `__cluster_metadata` topic was considered as a way to eliminate the credential island problem — the proxy could share the broker's own SCRAM credentials without separate provisioning. This was rejected because Kafka does not expose `__cluster_metadata` as a consumable topic, and the Admin API (`DescribeUserScramCredentials`) deliberately does not return the credential material (salt, serverKey, storedKey). This is by design: Kafka treats SCRAM credential material as write-only. There is no public API through which the proxy could obtain the credentials needed to perform SCRAM authentication. -### Using @Plugin for mechanism handlers +### Pluggable mechanism support -Making `MechanismHandlerFactory` a user-facing plugin (with `@Plugin` annotation and plugin discovery) was considered. This was rejected because: -- Mechanism handlers are internal implementation details, not user-facing extension points. -- Users configure _mechanisms_, not _handlers_. The mapping from mechanism name to handler is an implementation concern. -- `ServiceLoader` discovery is sufficient for internal extensibility. +Making mechanism support a user-facing plugin (with `@Plugin` annotation and plugin discovery) was considered. This was rejected because: +- Mechanism support is an internal implementation detail, not a user-facing extension point. +- Users configure _mechanisms_, not implementation classes. +- A small number of secure, high-quality implementations — one per mechanism — is easier to audit for correctness and security than an open plugin model where arbitrary implementations could be dropped on the classpath. ### Extending the OAUTHBEARER validation filter @@ -898,7 +729,7 @@ Adding SASL termination support to the existing OAUTHBEARER validation filter wa Supporting SASL PLAIN was deferred because: - PLAIN transmits passwords in cleartext (Base64 encoded, not encrypted), making it unsuitable for production use without TLS. - SCRAM provides mutual authentication and never transmits the password (though should also be used with TLS to avoid MitM attacks). -- If PLAIN support is needed in the future, it could be added as a new `MechanismHandler` implementation. +- If PLAIN support is needed in the future, it could be added as a new mechanism within the filter. ### GSSAPI (Kerberos) mechanism support @@ -906,7 +737,7 @@ Supporting SASL GSSAPI was deferred because: - GSSAPI/Kerberos requires the proxy to hold a service principal keytab and participate in the Kerberos infrastructure (KDC, realm trust, service tickets). This is a fundamentally different operational model from the credential store or JWKS endpoint approaches used by SCRAM and OAUTHBEARER. - Terminating Kerberos at the proxy would require the proxy to impersonate the broker's service principal (or hold its own), raising complex delegation and trust questions. - The demand for Kerberos termination (as opposed to passthrough) is lower than for SCRAM and OAUTHBEARER, which cover the most common credential isolation and identity provider integration use cases. -- If GSSAPI support is needed in the future, it could be added as a new `MechanismHandler` implementation, but the operational and trust model would need careful design. +- If GSSAPI support is needed in the future, it could be added as a new mechanism within the filter, but the operational and trust model would need careful design. ## References From 3d56a139bd5857be772548d5a8acdbdca7811d72 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:39:15 +0000 Subject: [PATCH 46/52] docs(proposal): add explicit security posture statement State upfront that the proposal adopts current security guidance (NIST SP 800-63B, OWASP) rather than matching Kafka's historical defaults. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 9fa88916..c0c0f9c1 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -59,6 +59,8 @@ It also aims to be flexible, so as to allow other mechanisms to be supported in Validation of legal compositions of SASL-related filters within a filter chain is not in scope for this proposal. +Because SASL termination places the proxy in the authentication decision path, this proposal adopts current security guidance (NIST SP 800-63B, OWASP) rather than matching Apache Kafka's historical defaults. This means, for example, higher PBKDF2 iteration counts for SCRAM and minimum password lengths that exceed Kafka's requirements, and not supporting SASL PLAIN at this time. + The proposal is organized per-component. Each component section covers its summary, API surfaces, configuration, threats and mitigations, and known limitations. ### Component overview From bfb393ab54c5c58c9cf979f6f439604fcaa70004 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:39:30 +0000 Subject: [PATCH 47/52] docs(proposal): add OAUTHBEARER error message leakage threat JWT validation failure details (expired token, wrong audience, wrong issuer) must not be returned to the client. Use a generic "Authentication failed" message, matching the SCRAM approach. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index c0c0f9c1..e6f6e607 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -343,6 +343,7 @@ The OAUTHBEARER mechanism is configured within the filter's `mechanisms` list wi |--------|------------| | Token from wrong audience or issuer -- a JWT issued for a different service or identity provider is presented to the proxy. | Both `expectedAudience` and `expectedIssuer` are required fields. Tokens that do not match are rejected. | | JWKS endpoint compromise -- an attacker controls the JWKS endpoint and supplies signing keys for forged tokens. | Mitigated operationally: the JWKS endpoint URL is set by the proxy administrator, not by clients. TLS protects the endpoint in transit (using the JVM's default trust store). | +| Leaking the reason for validation failure to the client, enabling probing by attackers. | On authentication failure, the filter returns a generic `"Authentication failed"` error message to the client, identical to the SCRAM approach. Detailed validation failure reasons are logged server-side for operator diagnostics but not included in the client-facing error response. | #### Known limitations From 63450f1ea56123c39f082548d7186656ef957682 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:39:50 +0000 Subject: [PATCH 48/52] docs(proposal): document OAUTHBEARER system property dependency Kafka's OAuthBearerValidatorCallbackHandler requires the oauthbearer.allowed.urls system property to be set. This is a JVM-global side effect that must be called out as a Kafka internal API dependency. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index e6f6e607..a3820384 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -679,6 +679,8 @@ This implementation uses several Kafka APIs that are not part of the [published **`org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslServerProvider`** — Called once (`initialize()`) to register the OAUTHBEARER SASL mechanism with the JVM's security provider infrastructure. The existing OAUTHBEARER validation filter uses this in the same way. There is no public API alternative. [Proposal 116][proposal-116] would copy this into the Kroxylicious namespace, giving stability control, but the functional dependency on Kafka's JSSE provider registration code remains. +**`org.apache.kafka.sasl.oauthbearer.allowed.urls`** — Kafka's `OAuthBearerValidatorCallbackHandler` checks this system property to restrict which JWKS endpoint URLs are permitted. The filter must set this system property at initialization time to include the configured `jwksEndpointUrl`. This is a JVM-global side effect — multiple filter instances with different JWKS endpoints must all contribute to the same system property. This is an inherent limitation of integrating with Kafka's callback handler, which was designed for broker-level (singleton) configuration, not per-filter-instance configuration. + **`org.apache.kafka.common.security.scram.internals.ScramMechanism`** — An enum identifying SCRAM-SHA-256 and SCRAM-SHA-512. Used internally by the SCRAM mechanism support and the keystore credential manager. There is no public API equivalent. [Proposal 116][proposal-116] would own this type, but it is a trivial enum that could equally be replaced with a Kroxylicious-native type. **`org.apache.kafka.common.security.scram.internals.ScramFormatter`** — Used by `KeystoreCredentialManager` to derive salted passwords, server keys, and stored keys from plaintext passwords. This is the only implementation of SCRAM key derivation available in the Kafka client library. There is no public API equivalent. [Proposal 116][proposal-116] would copy this into the Kroxylicious namespace, but unlike the protocol data classes, `ScramFormatter` is a functional security implementation (PBKDF2, HMAC) — the maintenance burden of keeping it current remains. From 531113956e8d019b5852848ce47289f79a448dac Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:40:34 +0000 Subject: [PATCH 49/52] docs(proposal): make PBKDF2 iteration count configurable Add --iterations option to add-user and update-password CLI commands (default 10,000, minimum 4,096). Update list-users to show mechanism and iteration count for credential auditing. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index a3820384..47318e0e 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -502,7 +502,7 @@ Create a new, empty KeyStore file. | `-t`, `--type` | No | `PKCS12` | KeyStore type (`PKCS12`, `JKS`). | ``` -keystore-credential-tool add-user -k -u [-p ] [-w ] [-m ] +keystore-credential-tool add-user -k -u [-p ] [-w ] [-m ] [-i ] ``` Add a SCRAM credential for a user. If the user already exists, their credential is replaced. @@ -514,6 +514,7 @@ Add a SCRAM credential for a user. If the user already exists, their credential | `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | | `-w`, `--user-password` | No | interactive prompt | User's password. Requires `--unlock-insecure-options`. | | `-m`, `--mechanism` | No | `SCRAM_SHA_256` | SCRAM mechanism (`SCRAM_SHA_256`, `SCRAM_SHA_512`). | +| `-i`, `--iterations` | No | `10000` | PBKDF2 iteration count. Minimum 4,096 ([RFC 5802][rfc5802]). Higher values increase brute-force resistance but also increase client-side authentication latency. | ``` keystore-credential-tool remove-user -k -u [-p ] @@ -528,7 +529,7 @@ Remove a user's credential from the KeyStore. | `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | ``` -keystore-credential-tool update-password -k -u [-p ] [-w ] [-m ] +keystore-credential-tool update-password -k -u [-p ] [-w ] [-m ] [-i ] ``` Update a user's password. Recomputes the SCRAM credential with a new salt. @@ -540,12 +541,13 @@ Update a user's password. Recomputes the SCRAM credential with a new salt. | `-p`, `--password` | No | interactive prompt | KeyStore password. Requires `--unlock-insecure-options`. | | `-w`, `--new-password` | No | interactive prompt | New password for the user. Requires `--unlock-insecure-options`. | | `-m`, `--mechanism` | No | `SCRAM_SHA_256` | SCRAM mechanism (`SCRAM_SHA_256`, `SCRAM_SHA_512`). | +| `-i`, `--iterations` | No | `10000` | PBKDF2 iteration count. Minimum 4,096 ([RFC 5802][rfc5802]). Higher values increase brute-force resistance but also increase client-side authentication latency. | ``` keystore-credential-tool list-users -k [-p ] ``` -List all usernames in the KeyStore. +List all credentials in the KeyStore. Output includes the username, SCRAM mechanism, and PBKDF2 iteration count for each entry, enabling operators to audit existing credentials. | Option | Required | Default | Description | |--------|----------|---------|-------------| @@ -557,7 +559,7 @@ List all usernames in the KeyStore. **Security measures:** - Passwords are read via interactive console prompts by default because passing secrets via CLI arguments is insecure (they appear in shell history and process listings). Command-line password arguments are supported but gated behind an `--unlock-insecure-options` flag that displays security warnings. - A 12-character minimum password length is enforced, following [NIST SP 800-63B][nist-sp800-63b] guidance. -- SCRAM credentials are generated with 10,000 PBKDF2 iterations and 20 bytes of random salt. The [RFC 5802][rfc5802] minimum is 4,096 (which is also the Kafka broker default). The [OWASP Password Storage Cheat Sheet][owasp-password-storage] currently recommends 600,000 iterations for PBKDF2-HMAC-SHA256, but that guidance targets password storage hashing where derivation happens once at write time. In SCRAM, the client performs the derivation on every authentication, so the iteration count directly affects authentication latency. 10,000 provides a reasonable balance between brute-force resistance and authentication performance for Kafka's typically long-lived connections. +- SCRAM credentials are generated with a configurable PBKDF2 iteration count (default 10,000, minimum 4,096) and 20 bytes of random salt. The [RFC 5802][rfc5802] minimum is 4,096 (which is also the Kafka broker default). The [OWASP Password Storage Cheat Sheet][owasp-password-storage] currently recommends 600,000 iterations for PBKDF2-HMAC-SHA256, but that guidance targets password storage hashing where derivation happens once at write time. In SCRAM, the client performs the derivation on every authentication, so the iteration count directly affects authentication latency. The default of 10,000 provides a reasonable balance between brute-force resistance and authentication performance for Kafka's typically long-lived connections. Operators can increase the iteration count for higher-security environments at the cost of longer client authentication times. - On POSIX systems, newly created KeyStore files are set to owner-only permissions (`rw-------`). When loading an existing KeyStore for modification (`add-user`, `remove-user`, `update-password`, `list-users`), the tool checks that the file does not have group or world read/write permissions and refuses to proceed if it does. #### Threats and mitigations From 1a38244a99562dfcb3784e582b2b15b2697d6816 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:40:54 +0000 Subject: [PATCH 50/52] docs(proposal): add offline brute-force threat to credential store Document the risk of offline password recovery after credential store exfiltration, with mitigations: configurable iteration count, file permission checks, and KeyStore encryption for in-transit protection. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 47318e0e..9cb876b2 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -567,6 +567,7 @@ List all credentials in the KeyStore. Output includes the username, SCRAM mechan | Threat | Mitigation | |--------|------------| | KeyStore file exposure -- an attacker gains read access to the KeyStore file on disk. | POSIX file permission check: the provider checks file permissions before loading, requiring `0600` or stricter by default. On Kubernetes/OpenShift where group-readable files are necessary, the `KROXYLICIOUS_DANGEROUSLY_CHANGE_PERMISSION_CHECK` environment variable allows relaxing to `0640`. The KeyStore itself is password-encrypted. | +| Offline brute-force after credential store theft -- an attacker who exfiltrates the KeyStore file attempts to recover passwords by brute-forcing the stored SCRAM credentials offline. | The PBKDF2 iteration count (configurable, default 10,000) is the primary defence against offline brute-force recovery of passwords from stolen credentials. The file permission checks reduce the likelihood of exfiltration in the first place. Operators in higher-security environments can increase the iteration count via the `--iterations` CLI option. The KeyStore itself is password-encrypted, which provides limited protection if stolen from the proxy host (where the password must also be available), but does protect credentials in transit provided the password is communicated out-of-band. | **Accepted risk: credential material in JVM heap.** SCRAM credential data (serverKey, storedKey, salt) is held in memory for the lifetime of the proxy. An attacker who can obtain a heap dump (e.g. via JMX, `/proc//mem`, or a core dump) can extract this material. There is no practical mitigation within a JVM. Operators should protect heap dump access through operational controls (JMX authentication, file permissions on core dumps, container security policies). From b6ca860a5de5c6c85f1e074e29966b80e7be95ff Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:41:36 +0000 Subject: [PATCH 51/52] docs(proposal): clarify fixedAuthDelay as floor, add SCRAM non-fail-fast Clarify that fixedAuthDelay is a minimum duration (floor), not an additive delay. For SCRAM, unknown users must not fail fast on the first round but continue into the second round to prevent round-trip-based user enumeration. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 9cb876b2..297902cb 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -214,7 +214,7 @@ The filter is configured via `SaslTerminationConfig`: |--------|------|----------|---------|-------------| | `mechanisms` | `List` | Yes | -- | List of mechanism configurations. Each entry includes a `mechanism` field (the IANA-registered mechanism name, e.g. `SCRAM-SHA-256`, `OAUTHBEARER`) and mechanism-specific configuration. At least one entry is required. | | `maxTimeBeforeReauth` | `Duration` | No | disabled | Maximum session lifetime before reauthentication is required (KIP-368). Uses golang-style duration syntax (e.g. `1h`, `30m`, `1h30m`). Omit or set to `0` to disable. | -| `fixedAuthDelay` | `Duration` | No | `200ms` | Fixed delay applied to all authentication rounds to prevent timing side-channel attacks that could enable user enumeration. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. | +| `fixedAuthDelay` | `Duration` | No | `200ms` | Minimum duration for each authentication round, enforced as a floor to prevent timing side-channel attacks that could enable user enumeration. If the real authentication work completes in less time, the response is held until the delay elapses. Set to `0` to disable if the deployment's threat model does not require user enumeration protection. | | `subjectBuilder` | `SaslSubjectBuilderService` | No | `DEFAULT_SUBJECT_BUILDER` | Plugin for constructing the `Subject` from authentication results. Defaults to `DEFAULT_SUBJECT_BUILDER`, consistent with the existing SASL inspection filter. | The `mechanisms` list elements are polymorphic. Jackson name-based deserialization (`@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "mechanism")`) resolves the concrete type from the `mechanism` field, which is the IANA-registered mechanism name (e.g. `SCRAM-SHA-256`, `OAUTHBEARER`). @@ -292,7 +292,7 @@ SCRAM mechanisms are configured within the filter's `mechanisms` list. The SCRAM | Threat | Mitigation | |--------|------------| | Username enumeration -- an attacker distinguishes existing from non-existing users by observing different error messages. | When a user is not found, the filter returns a generic `"Authentication failed"` error message identical to the message returned for incorrect credentials. | -| Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter applies a configurable fixed delay (`fixedAuthDelay`) to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. | +| Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter enforces `fixedAuthDelay` as a minimum duration (floor) for each authentication round. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. Additionally, when a user is not found the filter must not fail fast on the first round but must continue into the second round, so that attackers cannot distinguish existing from non-existing users by the number of round-trips. | | SCRAM protocol correctness -- a bug in the SCRAM implementation could allow authentication bypass or credential leakage. | Delegated to Kafka's own `SaslServer`, which is widely deployed and well-tested. The filter is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. | #### Known limitations @@ -610,7 +610,9 @@ When a user is not found in the credential store, the handler returns a generic ### Timing side-channel mitigation -Without mitigation, an attacker could distinguish existing from non-existing users by measuring response times: credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists. Rather than trying to equalize these inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter applies a configurable fixed delay (`fixedAuthDelay`) to all authentication rounds. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. +Without mitigation, an attacker could distinguish existing from non-existing users by measuring response times: credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists. Rather than trying to equalize these inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter enforces `fixedAuthDelay` as a minimum duration (floor) for each authentication round. If the real authentication work completes in less time, the response is held until the delay elapses. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. + +Additionally, for SCRAM, when a user is not found in the credential store the filter must not fail fast on the first round but must continue the exchange into the second round, so that attackers cannot distinguish existing from non-existing users by the number of round-trips. ### Observability From 7725263a7201c860cd9bbc45c0fd8b6032a10e68 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 3 Aug 2026 02:42:11 +0000 Subject: [PATCH 52/52] docs(proposal): add input size limits for authBytes and username Add per-mechanism authBytes limits (SCRAM: 4KB, OAUTHBEARER: 128KB) and a 255-character username length limit enforced in both the SCRAM handler and CLI tool. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/124-sasl-termination.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/124-sasl-termination.md b/proposals/124-sasl-termination.md index 297902cb..334aab9d 100644 --- a/proposals/124-sasl-termination.md +++ b/proposals/124-sasl-termination.md @@ -247,6 +247,7 @@ filters: |--------|------------| | Unauthenticated request bypass -- a client sends Kafka protocol requests (Produce, Fetch, etc.) before completing SASL authentication. | The security barrier rejects all non-SASL request types until the state reaches `Authenticated`. Rejected requests receive `SASL_AUTHENTICATION_FAILED` and the connection is closed immediately. | | Session expiry evasion -- an authenticated client continues sending requests after its session has expired without reauthenticating. | On every non-SASL request in the `Authenticated` state, the filter checks whether the session has expired. If so, the request is rejected with `SASL_AUTHENTICATION_FAILED` and the connection is closed. `SASL_HANDSHAKE` / `SASL_AUTHENTICATE` are always permitted, allowing reauthentication. | +| Oversized `SaslAuthenticate` payload -- a client sends an excessively large `authBytes` field in a `SaslAuthenticateRequest`, consuming memory or processing time. | The filter enforces per-mechanism upper bounds on the size of `authBytes`. For SCRAM: 4KB (legitimate SCRAM messages are at most a few hundred bytes — username up to 255 characters plus fixed-size nonce and proof fields). For OAUTHBEARER: 128KB (accommodates large enterprise JWT tokens with many group/role claims while preventing multi-MB allocations). Payloads exceeding the limit are rejected and the connection is closed. | #### Known limitations @@ -293,6 +294,7 @@ SCRAM mechanisms are configured within the filter's `mechanisms` list. The SCRAM |--------|------------| | Username enumeration -- an attacker distinguishes existing from non-existing users by observing different error messages. | When a user is not found, the filter returns a generic `"Authentication failed"` error message identical to the message returned for incorrect credentials. | | Timing side-channel -- an attacker distinguishes existing from non-existing users by measuring response times (credential lookup, deserialization, and SCRAM server creation take different amounts of time depending on whether the user exists). | Rather than trying to equalize inherently different code paths (which is fragile under JIT optimizations and varies by credential store implementation), the filter enforces `fixedAuthDelay` as a minimum duration (floor) for each authentication round. The delay is long enough to swamp any timing differences but short enough to be negligible for Kafka's typically long-lived connections. If the observed authentication duration exceeds the configured delay, a WARN log is emitted indicating the delay should be increased. The delay can be disabled by setting `fixedAuthDelay` to `0` if the deployment's threat model does not require user enumeration protection. Additionally, when a user is not found the filter must not fail fast on the first round but must continue into the second round, so that attackers cannot distinguish existing from non-existing users by the number of round-trips. | +| Oversized username -- a client sends an excessively long username in the SCRAM client-first-message, consuming storage or processing resources. | The filter enforces a maximum username length of 255 characters, matching POSIX `LOGIN_NAME_MAX` and accommodating all common username formats (simple names, email addresses, short DNs). The same limit is enforced by the CLI tool at credential creation time. | | SCRAM protocol correctness -- a bug in the SCRAM implementation could allow authentication bypass or credential leakage. | Delegated to Kafka's own `SaslServer`, which is widely deployed and well-tested. The filter is responsible only for credential lookup and passing credentials to the `SaslServer` via a `CallbackHandler`. | #### Known limitations