From 375d0fd4f8aa65b5cc8e9533658a62d5d6d09def Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mateusz=20Ko=C5=82odziejczyk?= Date: Thu, 6 Aug 2026 19:04:08 +0200 Subject: [PATCH 1/2] [RORDEV-2145] Document PKI authentication Adds the pkis connector, the pki_authentication / pki_authorization / pki_auth rules, and brings the encryption section up to date: client_authentication is no longer a boolean but none / optional / required. The connector section leads with what stops PKI working rather than with syntax, because all three failure modes are silent. TLS terminated at a load balancer means no certificate ever reaches the node; client_authentication left at none means the node never asks for one; and ReadonlyREST cannot warn about either, since the ACL has no view of the TLS configuration. It also warns that PKI is only as strong as the verification the TLS layer performs. verification_mode: none makes Elasticsearch request a certificate and validate nothing, so anyone who can run a CA can issue one carrying a service's CN and be authenticated as that service - which is the strongest argument for pinning issuer_dn. The Active Directory case is called out early: the login identity is the UPN and is absent from the DN entirely, so a CN-based provider yields an unusable username and no pattern can recover it. --- elasticsearch.md | 282 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 280 insertions(+), 2 deletions(-) diff --git a/elasticsearch.md b/elasticsearch.md index efabfda..a713319 100644 --- a/elasticsearch.md +++ b/elasticsearch.md @@ -473,14 +473,29 @@ readonlyrest.ssl.key_alias: "my-server-cert" # optional; if omitted, RO readonlyrest.ssl.allowed_protocols: [TLSv1.2, TLSv1.3] readonlyrest.ssl.allowed_ciphers: [TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256] -# Optional: mutual TLS — require clients to present a certificate -readonlyrest.ssl.client_authentication: false # default +# Optional: mutual TLS — ask clients to present a certificate +readonlyrest.ssl.client_authentication: none # none (default) | optional | required # Optional: custom trust anchor for client certificates (defaults to JVM truststore) readonlyrest.ssl.truststore_file: "truststore.jks" readonlyrest.ssl.truststore_pass: "" ``` +`client_authentication` accepts: + +| Value | Meaning | +| --- | --- | +| `none` | No certificate is requested. The default. | +| `optional` | A certificate is requested and verified if presented; clients without one still connect. | +| `required` | A certificate is mandatory; the TLS handshake fails without a valid one. | + +Use `optional` when certificate-bearing services and password-bearing users share the same port. Use +`required` when every client holds a certificate — it moves the rejection from the ACL to the handshake, +so a client without one sees a TLS error rather than an HTTP response. + +Once clients present certificates, the [`pkis` connector](elasticsearch.md#pki-connector) can turn them +into ReadonlyREST users, so a service authenticates with nothing but its certificate. + **PEM option (preferred):** ```yaml http.type: ssl_netty4 @@ -1207,6 +1222,89 @@ See the dedicated [LDAP section](elasticsearch.md#ldap-connector) * Groups logic syntax can be uses as part of this rule, as described in the [Checking groups logic section](details/authorization-rules-details.md#checking-groups-logic) * For more information on the ROR's authorization rules, see [Authorization rules details](details/authorization-rules-details.md) +##### `pki_authentication` + +Authenticates a client by the TLS client certificate its connection was established with, using the +configured PKI provider (here `corporate_pki`). See the [PKI connector section](elasticsearch.md#pki-connector) +to learn how to configure the provider. + +simple version: +`pki_authentication: corporate_pki` + +extended version: +```yaml +pki_authentication: + name: corporate_pki + users: ["svc-logstash", "beats-*"] # optional: only accept these usernames +``` + +- The certificate is the only credential. No password, token or header is involved. +- The certificate has already been verified by whichever component terminated TLS, so this rule never + decides whether a certificate is trustworthy — only who it identifies. +- A request that arrives **without** a certificate does not match, and evaluation moves on to the next + block. That is what lets certificate-bearing services and password-bearing users share one port. + +##### `pki_authorization` + +```yaml +pki_authorization: + name: "corporate_pki" + groups_any_of: ["ingest_services"] +``` + +- It reads the groups out of the client certificate, using the `groups` section of the configured PKI + provider. The provider must define one, or the settings are rejected at startup. +- The groups are *external* groups, so they flow into the usual + [external to local groups mapping](details/groups-rule-mapping.md). Values that no mapping matches + are simply discarded, which is how you keep certificate attributes that are not roles (`OU=Warsaw`, + `OU=IT`) out of ReadonlyREST. +- Groups logic syntax can be used as part of this rule, as described in the + [Checking groups logic section](details/authorization-rules-details.md#checking-groups-logic) + +##### `pki_auth` + +Shorthand rule that combines `pki_authentication` and `pki_authorization` together. + +```yaml +pki_auth: + name: "corporate_pki" + groups_any_of: ["ingest_services"] +``` + +The same can be written as: + +```yaml +pki_authentication: corporate_pki +pki_authorization: + name: "corporate_pki" + groups_any_of: ["ingest_services"] +``` + +The most common enterprise arrangement, however, takes the identity from the certificate and the groups +from the corporate directory: + +```yaml +- name: "Platform services" + pki_authentication: + name: "corporate_pki" + ldap_authorization: + name: "corporate_ldap" + groups_any_of: ["platform_services"] + indices: ["logs-*"] +``` + +**⚠️IMPORTANT** for this to work, the username the certificate yields has to be the one the directory +can look up. If your CA puts a display name in the `CN` but LDAP keys on `uid` or `sAMAccountName`, the +LDAP search fails with a confusing "user not found". See +[reading the identity](elasticsearch.md#reading-the-identity) for how to extract the right value without +re-issuing certificates. + +* [Impersonation](details/impersonation.md) works as it does for `proxy_auth`: when the rule declares a + `users` list the impersonated user is checked against it, and without one impersonation is refused for + that rule. Certificate-derived groups can never be impersonated, because an impersonator presents no + certificate belonging to the impersonated user. +* For more information on the ROR's authorization rules, see [Authorization rules details](details/authorization-rules-details.md) + ##### `jwt_authentication` See below, the dedicated [JSON Web Tokens section](elasticsearch.md#json-web-token-jwt-auth). It's an authentication rule. @@ -2723,6 +2821,186 @@ readonlyrest: search_groups_base_DN: "ou=Groups,dc=example2,dc=com" ``` +#### PKI connector + +A PKI provider turns a TLS client certificate into a ReadonlyREST user. It is how a service +authenticates with nothing but the certificate it already holds — no password in a `logstash.yml`, no +token on disk. + +Providers are declared in the `pkis` section and referenced by name from the +[`pki_authentication`](elasticsearch.md#pki_authentication), +[`pki_authorization`](elasticsearch.md#pki_authorization) and +[`pki_auth`](elasticsearch.md#pki_auth) rules. + +```yaml +readonlyrest: + + access_control_rules: + + - name: "Logstash ingest" + pki_authentication: + name: "corporate_pki" + users: ["logstash-*"] + actions: ["indices:data/write/*"] + indices: ["ingest-*"] + + pkis: + - name: corporate_pki + users: + user_id_attribute: "CN" # the default; shown for clarity +``` + +##### Before you start + +ReadonlyREST can only read a certificate that was presented to **this** Elasticsearch node, so: + +- **TLS has to terminate at Elasticsearch.** If a load balancer, ingress or service mesh terminates it + upstream, no certificate reaches ReadonlyREST and PKI rules never match. This is the single most common + reason PKI "does not work". Such deployments have to move TLS termination, or keep using `proxy_auth`. +- **Client authentication has to be on.** Either `readonlyrest.ssl.client_authentication` (see + [Encryption](elasticsearch.md#encryption)) or `xpack.security.http.ssl.client_authentication`, set to + `optional` or `required`. With `none` the node never asks for a certificate, so PKI rules never match. + ReadonlyREST cannot detect this for you: the ACL has no view of your TLS configuration. +- **Elasticsearch 7.0 or newer.** On 6.7 a request cannot be traced back to the connection it arrived on. + +**⚠️IMPORTANT** PKI is only as strong as the verification your TLS layer performs. A setting such as +`xpack.security.http.ssl.verification_mode: none` makes Elasticsearch request a certificate and then +validate nothing, so a certificate from *any* CA is accepted — and anyone able to run a CA can issue one +carrying `CN=svc-logstash` and be authenticated as that service. ReadonlyREST cannot tell the difference: +by the time a rule sees the certificate it looks legitimate. Always verify chains, and consider pinning +`issuer_dn` (below), which is the one constraint a forged subject cannot satisfy. + +##### Reading the identity + +The `users` section says how to get a username out of a certificate. It follows the same shape as the +LDAP connector: an optional `mode` selecting a variant, and a default that needs no `mode` at all. + +| `mode` | Keys | What it does | +| --- | --- | --- | +| `subject_dn_attribute` *(default)* | `user_id_attribute` | Reads a named attribute out of the subject DN. Defaults to `CN`. | +| `subject_dn_pattern` | `pattern` | Applies a regular expression to the subject DN. | +| `san` | `san_type`, optional `pattern` | Reads a Subject Alternative Name entry. | + +Attribute names are matched case-insensitively, and the DN is parsed structurally rather than as a +string, so escaping and multi-valued RDNs are handled for you: `CN=Smith\, John` yields `Smith, John`. + +**Active Directory.** In AD the login identity is the userPrincipalName, not the `CN` — a certificate +with `CN=John Smith` and UPN `jsmith@corp.example.com` gives an unusable username under `CN`, and the +login name is not in the DN at all. Read the UPN instead: + +```yaml + pkis: + - name: ad_pki + users: + mode: san + san_type: upn # the otherName entry with OID 1.3.6.1.4.1.311.20.2.3 +``` + +`san_type` accepts `dns`, `email`, `uri`, `ip` and `upn`. Where a certificate carries several entries of +the requested type — routine for host certificates, which often hold an FQDN, a short name and aliases — +the first is used, and an optional `pattern` selects among them: + +```yaml + pkis: + - name: host_pki + users: + mode: san + san_type: dns + pattern: "^(.+)\\.corp\\.example\\.com$" +``` + +For a DN your CA lays out unusually, the regular expression escape hatch applies to the RFC 2253 +rendering of the subject, and captured values are unescaped for you: + +```yaml + pkis: + - name: legacy_pki + users: + mode: subject_dn_pattern + pattern: "^CN=([^,]+),OU=Service Accounts,.*$" +``` + +A pattern has to compile, define exactly one capture group, and be unable to match an empty value — +all checked when the settings load, not when a request arrives. + +##### Reading groups + +An optional `groups` section makes the certificate carry authorization too, which suits clusters with no +directory to consult: + +```yaml + pkis: + - name: internal_pki + users: + user_id_attribute: "CN" + groups: + group_id_attribute: "OU" +``` + +The same three modes are available. Where a username takes the first value found, groups take all of +them — so one `subject_dn_pattern` serves both, and `pattern: "OU=grp-([^,]+)"` yields a group per match. + +Note that a DN's OUs are frequently a position in an org tree rather than a set of roles: + +``` +CN=beats-01,OU=ingest,O=Corp -> ["ingest"] +CN=jsmith,OU=Engineering,OU=EMEA,OU=Employees -> ["Engineering", "EMEA", "Employees"] +``` + +ReadonlyREST does not interpret their order or hierarchy, and offers no filter — mapping only the +role-bearing ones is your decision, expressed in the +[groups mapping](details/groups-rule-mapping.md). Anything you do not map is discarded. + +Group *names* are not supported: a certificate carries a single value per attribute, so there is no +second attribute to read a display name from. `group_name_attribute` is rejected at startup rather than +silently ignored. + +##### Restricting which certificates a provider speaks for + +One corporate CA usually issues to more than one population: + +``` +CN=svc-logstash,OU=Services,DC=corp,DC=example,DC=com <- machine +CN=John Smith,OU=People,DC=corp,DC=example,DC=com <- human +``` + +Both chain to the same CA, so TLS trusts them equally and a plain `CN` extractor would authenticate +both. Two optional constraints keep them apart. They are checked before anything is extracted, so a +certificate out of scope yields neither a username nor groups. + +```yaml + pkis: + - name: services_pki + subject_dn_base: "OU=Services,DC=corp,DC=example,DC=com" # subject must end with this + issuer_dn: "CN=Corp Issuing CA,DC=corp,DC=example,DC=com" # issuer must match exactly + users: + user_id_attribute: "CN" + + - name: employees_pki + subject_dn_base: "OU=People,DC=corp,DC=example,DC=com" + users: + user_id_attribute: "CN" +``` + +`issuer_dn` covers what `subject_dn_base` cannot: any CA your node trusts can mint any name, so where a +truststore holds more than one CA, pinning the issuer is what stops one CA impersonating identities from +another. + +##### Migrating from the X-Pack PKI realm + +The concepts line up. X-Pack derives a username from the subject DN with `username_pattern` and resolves +roles through role mappings; ReadonlyREST derives one with `mode`/`user_id_attribute`/`pattern` and +resolves groups from the `users` section, LDAP, or the certificate itself. An X-Pack `username_pattern` +regex can be carried over as-is under `mode: subject_dn_pattern`. + +##### Notes + +- Certificate revocation (CRL/OCSP) is not consulted — Elasticsearch does not support it either. To + deprovision a compromised identity, remove it from the `users` section or from a rule's `users` list + and reload the settings. +- The certificate belongs to the connection, so every request multiplexed over one keep-alive connection + carries the same identity. Settings reloads apply to new connections. + #### External Basic Auth ReadonlyREST will forward the received `Authorization` header to a website of choice and evaluate the returned HTTP status code to verify the provided credentials. This is useful if you already have a web server with all the credentials configured and the credentials are passed over the `Authorization` header. From 64cef56743f6adbcdef6529b910badee4cd9b846 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mateusz=20Ko=C5=82odziejczyk?= Date: Fri, 7 Aug 2026 11:29:55 +0200 Subject: [PATCH 2/2] Review fixes --- elasticsearch.md | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/elasticsearch.md b/elasticsearch.md index a713319..039938d 100644 --- a/elasticsearch.md +++ b/elasticsearch.md @@ -588,12 +588,13 @@ readonlyrest.ssl_internode.hostname_verification: true By default, the server does not request a client certificate. When enabled, Elasticsearch verifies the client's identity via mutual TLS. -For external REST API: +For the external REST API, the setting takes `none`, `optional` or `required` — see the value table under +[External REST API](elasticsearch.md#external-rest-api): ```yaml -readonlyrest.ssl.client_authentication: true +readonlyrest.ssl.client_authentication: required ``` -For internode communication: +For internode communication it is a boolean: ```yaml readonlyrest.ssl_internode.client_authentication: true ``` @@ -1242,7 +1243,9 @@ pki_authentication: - The certificate has already been verified by whichever component terminated TLS, so this rule never decides whether a certificate is trustworthy — only who it identifies. - A request that arrives **without** a certificate does not match, and evaluation moves on to the next - block. That is what lets certificate-bearing services and password-bearing users share one port. + block. That is what lets certificate-bearing services and password-bearing users share one port — + but only where client authentication is `optional`. With `required`, a client holding no certificate + is rejected during the TLS handshake and never reaches the ACL at all. ##### `pki_authorization` @@ -2867,8 +2870,10 @@ ReadonlyREST can only read a certificate that was presented to **this** Elastics `xpack.security.http.ssl.verification_mode: none` makes Elasticsearch request a certificate and then validate nothing, so a certificate from *any* CA is accepted — and anyone able to run a CA can issue one carrying `CN=svc-logstash` and be authenticated as that service. ReadonlyREST cannot tell the difference: -by the time a rule sees the certificate it looks legitimate. Always verify chains, and consider pinning -`issuer_dn` (below), which is the one constraint a forged subject cannot satisfy. +by the time a rule sees the certificate it looks legitimate. Always verify chains. Pinning `issuer_dn` +(below) is defence in depth, not a replacement: the issuer is a field *inside* the presented certificate, +so with chain validation disabled a forged certificate simply carries a matching one. It constrains +certificates your TLS layer has already verified — it cannot make an unverified one trustworthy. ##### Reading the identity @@ -2942,7 +2947,7 @@ them — so one `subject_dn_pattern` serves both, and `pattern: "OU=grp-([^,]+)" Note that a DN's OUs are frequently a position in an org tree rather than a set of roles: -``` +```text CN=beats-01,OU=ingest,O=Corp -> ["ingest"] CN=jsmith,OU=Engineering,OU=EMEA,OU=Employees -> ["Engineering", "EMEA", "Employees"] ``` @@ -2959,7 +2964,7 @@ silently ignored. One corporate CA usually issues to more than one population: -``` +```text CN=svc-logstash,OU=Services,DC=corp,DC=example,DC=com <- machine CN=John Smith,OU=People,DC=corp,DC=example,DC=com <- human ``` @@ -2996,10 +3001,14 @@ regex can be carried over as-is under `mode: subject_dn_pattern`. ##### Notes - Certificate revocation (CRL/OCSP) is not consulted — Elasticsearch does not support it either. To - deprovision a compromised identity, remove it from the `users` section or from a rule's `users` list - and reload the settings. + deprovision a compromised identity, remove it from the rule's `users` allowlist, or from the top-level + `users` definitions section so that it resolves no groups, and reload the settings. The provider's own + `users` section only says how to read a username out of a certificate — it grants nothing, so editing + it deprovisions nobody. - The certificate belongs to the connection, so every request multiplexed over one keep-alive connection - carries the same identity. Settings reloads apply to new connections. + carries the same identity. The ACL is still evaluated per request against the settings in force at that + moment, so a reload takes effect on the next request, existing connections included — there is no need + to close them. #### External Basic Auth