From 471d978e01b31e05819aa796f13e8e0f82616629 Mon Sep 17 00:00:00 2001 From: Denis Mishin Date: Fri, 31 Jul 2026 12:13:54 -0400 Subject: [PATCH 1/2] docs: document secret injection settings and add a guide --- content/docs/guides/secrets.mdx | 286 +++++++++++++++++ content/docs/reference/metrics.mdx | 21 ++ content/docs/reference/reference.json | 53 ++++ content/docs/reference/routes/headers.mdx | 30 +- content/docs/reference/secrets.mdx | 296 ++++++++++++++++++ .../examples/guides/secrets/config.yaml.md | 40 +++ .../guides/secrets/docker-compose.yaml.md | 42 +++ cspell.json | 2 + 8 files changed, 769 insertions(+), 1 deletion(-) create mode 100644 content/docs/guides/secrets.mdx create mode 100644 content/docs/reference/secrets.mdx create mode 100644 content/examples/guides/secrets/config.yaml.md create mode 100644 content/examples/guides/secrets/docker-compose.yaml.md diff --git a/content/docs/guides/secrets.mdx b/content/docs/guides/secrets.mdx new file mode 100644 index 000000000..8044f25d0 --- /dev/null +++ b/content/docs/guides/secrets.mdx @@ -0,0 +1,286 @@ +--- +title: Inject a Rotating Credential into Upstream Requests +sidebar_label: Secret Injection +lang: en-US +keywords: + [ + pomerium, + secrets, + secret injection, + credential rotation, + api token, + set request headers, + docker compose, + ] +description: Keep an upstream API token out of your Pomerium configuration. Pomerium reads it from a file, injects it into every upstream request, picks up rotations within seconds, and rejects requests when the credential cannot be read. +--- + +import Config from '/content/examples/guides/secrets/config.yaml.md'; +import Compose from '/content/examples/guides/secrets/docker-compose.yaml.md'; + +# Inject a Rotating Credential into Upstream Requests + +## What this guide does + +Many upstream services accept only a static credential: an API token, a shared basic-auth password, or an admin key. The usual way to front such a service with Pomerium is to write the credential straight into the route's [`set_request_headers`](/docs/reference/routes/headers#set-request-headers). That works, but the credential then lives in your Pomerium configuration, and every rotation becomes a configuration change. + +In this guide you move that credential out of the configuration. You put it in a file, name it with a [secret binding](/docs/reference/secrets#bindings), and reference the binding from the route. Pomerium reads the file in the background and substitutes the value on each request. You will then rotate the credential while the stack keeps running, and watch Pomerium reject requests when the credential is gone, instead of forwarding them without it. + +The stack is Pomerium Core plus a small echo service that stands in for your API and shows you exactly which headers it received. + +## When to use this guide + +Use it when an upstream credential is rotated by something other than you editing Pomerium's configuration: a secret manager syncing to disk, a Kubernetes `Secret` mounted into the container, a cron job, or an operator writing a new token. Use it also when you simply do not want the credential stored in the same file as your routes and policies. + +If the credential never changes and your configuration file is already the trusted place for it, a static header value is enough. Compare with the [AdGuard guide](/docs/guides/ad-guard), which injects a fixed basic-auth credential from the configuration file. + +The binding table is a Pomerium Core setting written in the configuration file. It is not available as an Ingress annotation, and the Pomerium Zero Console does not expose it. + +## How it works + +Pomerium's Authorize service builds the request headers while it evaluates the policy. It keeps the current value of each binding in memory, so injection adds no reads on the request path. + +```mermaid +sequenceDiagram + participant User + participant Pomerium + participant Secret as Secret file + participant Upstream + + Secret-->>Pomerium: background read, then re-read on change or refresh + User->>Pomerium: request, already signed in + Pomerium->>Pomerium: policy allows, build headers from current value + Pomerium->>Upstream: request with Authorization header + Upstream-->>User: response +``` + +Two properties matter for the rest of this guide: + +- **Rotation needs no reload.** Pomerium re-reads the file on a timer and also watches it for changes. +- **Injection fails closed.** If the value cannot be read, Pomerium answers `503 Service Unavailable` rather than sending the request without the header. + +The full behavior is described in the [Secrets reference](/docs/reference/secrets). + +## Prerequisites + +This guide assumes you've completed the [Quickstart](/docs/get-started/quickstart), so you already have Pomerium running and signing users in through the hosted authenticate service. + +You also need: + +- [Docker](https://docs.docker.com/install/) and [Docker Compose](https://docs.docker.com/compose/install/) +- A domain you control for the route (this guide uses `api.yourdomain.com`), with DNS pointing at the host +- Ports 80 and 443 reachable on that host, so `autocert` can obtain a certificate +- A credential to inject. Any string works for the walkthrough. + +The Compose file uses the `pomerium/pomerium:main` image tag, because secret injection is newer than the latest pinned release. Pin a released tag once one includes it. + +## Step 1: create the credential file + +Make a directory for credentials and write the token into it. Use `printf`, not `echo -n`, for predictable output: + +```bash +mkdir -p secrets +printf 'token-v1' > secrets/upstream-api-token +chmod 0600 secrets/upstream-api-token +``` + +Mode `0600` keeps the file private to its owner while still letting you rewrite it later in this guide. Pomerium only reads it. + +Pomerium removes exactly one trailing newline from the file, so a value written by `echo` or by a text editor works too. Any other whitespace is part of the value. + +## Step 2: configure Pomerium + +Create `config.yaml` next to the `secrets` directory: + + + +Replace `api.yourdomain.com` with your domain and `you@example.com` with your email. + +Three parts work together: + +- `secrets.bindings` gives the credential an ID, `upstream-api-token`, and a `file://` URL with an absolute path inside the container. +- `secrets.defaults` sets how often Pomerium re-reads the file (`refresh`) and how long it keeps using the last good value while re-reads fail (`stale_grace`). +- The route's `set_request_headers` references the binding as `${secret.upstream-api-token}`. + +The `stale_grace` of `5m` is short so that you can observe the fail-closed path later in this guide. For production, the [default of 30 minutes](/docs/reference/secrets#defaults) rides out a longer outage of whatever writes the file. + +## Step 3: define the stack + +Create `docker-compose.yaml`: + + + +Note the mount: the `secrets` **directory** is mounted, not the single file. A single-file bind mount is bound to one inode. When a tool replaces the credential, which is how most secret managers write files, the container keeps pointing at the old inode: the path inside the container stops resolving, Pomerium reports the secret as missing, and requests fail closed once the grace window ends. Mounting the directory avoids this. + +## Step 4: start the stack + +```bash +docker compose up -d +``` + +Wait for the certificate to be issued, then confirm Pomerium read the credential: + +```bash +docker compose logs pomerium | grep secret +``` + +Expected output: + +```json +{ + "level": "info", + "ref": "file:///etc/pomerium/secrets/upstream-api-token", + "label": "upstream-api-token", + "state": "fresh", + "message": "secret resolved" +} +``` + +Pomerium does not wait for this read at startup. If it fails, Pomerium still serves every other route, and only requests that need this credential are rejected. + +## Verify the setup + +1. **The route requires authentication.** In a fresh browser, open `https://api.yourdomain.com`. You are redirected to sign in. +2. **The credential reaches the upstream.** Sign in. The echo service returns the request it received. In the `headers` object you see: + + ```text + "authorization": "Bearer token-v1" + ``` + + The value came from the file, not from `config.yaml`. + +3. **Pomerium reports a healthy binding.** Metrics confirm the same thing without a browser: + + ```bash + curl -s http://127.0.0.1:9090/metrics | grep pomerium_secrets + ``` + + ```text + pomerium_secrets_cache_state{ref_label="upstream-api-token",state="fresh"} 1 + pomerium_secrets_fetches_total{outcome="success",provider="file"} 1 + pomerium_secrets_header_inject_total{outcome="injected",route_id="..."} 1 + ``` + + Your output also carries a `hostname` label on each series. `state="fresh"` means the last read succeeded, and `outcome="injected"` counts headers that were built with a credential. See [Secrets metrics](/docs/reference/metrics#secrets) for the full list. + +## Rotate the credential + +Now replace the value the way a secret manager would, by writing a new file and moving it into place: + +```bash +printf 'token-v2' > secrets/.upstream-api-token.new +chmod 0600 secrets/.upstream-api-token.new +mv secrets/.upstream-api-token.new secrets/upstream-api-token +``` + +Reload `https://api.yourdomain.com`. The echo service now reports: + +```text +"authorization": "Bearer token-v2" +``` + +The new value is usually visible about a second after the move. There is no configuration change, no `docker compose restart`, and no dropped request: requests during the switch carry either the old value or the new one, never an empty header. + +An in-place rewrite (`printf 'token-v3' > secrets/upstream-api-token`) works the same way. An atomic move is still the safer habit, because a reader can never observe a half-written file. + +## Confirm the fail-closed behavior + +Delete the credential and watch what Pomerium does: + +```bash +rm secrets/upstream-api-token +``` + +For roughly the next five minutes, the `stale_grace` window from your configuration, requests still succeed with the last good value, and Pomerium warns on the state change: + +```json +{ + "level": "warn", + "ref": "file:///etc/pomerium/secrets/upstream-api-token", + "label": "upstream-api-token", + "state": "stale", + "error_class": "not_found", + "message": "secret serving stale" +} +``` + +Then the value is dropped and the route starts answering `503 Service Unavailable`. Two details make the exact moment approximate: the window runs from the last successful read, which was up to one `refresh` interval before you deleted the file, and the drop happens on the next failed read attempt, which for a missing file is within 30 seconds. Expect the switch between about four and five and a half minutes after the deletion. + +Reload `https://api.yourdomain.com` in the browser where you are still signed in. Pomerium's error page reports `503 Service Unavailable`, with no detail about the credential. Check it from your signed-in session, not with a plain `curl`: an unauthenticated request is redirected to sign in as usual, because the policy decision comes first. + +The Pomerium log names the binding and the header that could not be filled: + +```json +{ + "level": "warn", + "binding": "upstream-api-token", + "header": "authorization", + "route": "https://api.yourdomain.com → http://upstream:8080", + "message": "authorize: secret unavailable, denying request" +} +``` + +This is the behavior to expect during a bad rotation. Pomerium does not forward the request with the header missing, which would look to the upstream like an anonymous call. + +Put the credential back and the route recovers with no restart: + +```bash +printf 'token-v2' > secrets/upstream-api-token +chmod 0600 secrets/upstream-api-token +``` + +Recovery takes up to `negative_ttl`, 30 seconds by default, because Pomerium stops reading a missing secret for that long after each failed attempt. This is slower than the rotation earlier in this guide, where the file was never missing. Recovery is logged as `secret recovered`. + +## Troubleshooting + +| Symptom | Likely cause | What to check | Fix | +| --- | --- | --- | --- | +| Pomerium exits at startup with `validation error ... unknown secret "..."` | A route references a binding ID that is not in the table | Compare the ID in `set_request_headers` with the keys under `secrets.bindings` | Correct the ID. IDs are case-sensitive and cannot contain dots | +| Every request to the route returns `503`, and the log says `secret unavailable` | Pomerium never read the file | `docker compose logs pomerium \| grep secret`. A missing file logs `secret not found; negative-caching`, which names the URL it tried to read | Check that the `url` path matches the container path in the volume mount, for example `./secrets:/etc/pomerium/secrets:ro` with `file:///etc/pomerium/secrets/upstream-api-token` | +| `503`, but the log shows no secret state change at all | The file exists and cannot be read, usually a permissions problem | `curl -s http://127.0.0.1:9090/metrics \| grep -E 'cache_state\|fetches_total'`. `cache_state{state="failed"}` means the binding has never been read; a growing `fetches_total{outcome="error"}` confirms a read error | Make the file readable by the user Pomerium runs as. Only state changes are logged, and a binding that never succeeds has no state change to log | +| Requests start failing right after a rotation, and the log says `secret not found` | The single file was bind-mounted instead of the directory, so replacing it left the container pointing at the old inode | The `volumes` entry in `docker-compose.yaml` | Mount the directory (`./secrets:/etc/pomerium/secrets:ro`) and recreate the container | +| The upstream returns `401` while Pomerium injects the header | The value carries characters you did not intend | `xxd secrets/upstream-api-token \| tail -2`. Pomerium strips one trailing newline, and nothing else | Rewrite the file with `printf '%s' "$TOKEN"`. A value containing a carriage return, a line feed, or a null byte is rejected when the header is built, so those requests get `503` instead | +| Only some routes return `503` | Expected. Rejection is per binding | Which bindings the failing routes reference | Fix the affected binding. Routes that reference healthy bindings are unaffected | + +The Pomerium image is distroless and has no shell, so `docker compose exec pomerium ls` will not work. Inspect the host directory and the volume definition instead. + +## Security notes + +- **The file is the boundary now.** Anyone who can read `secrets/upstream-api-token` on the host, or read the container's filesystem, has the credential. Keep it owned by a single user with a restrictive mode, and keep it out of version control and out of image builds. +- **The upstream still trusts the credential, not the user.** Any client that can present the token to the upstream directly bypasses Pomerium. Keep it off published ports and on an internal Docker network shared with Pomerium alone, as the Compose file does. +- **A route decides which binding it injects.** Bindings do not restrict which upstream receives a credential, so treat write access to routes as write access to the credential's use. The binding table is the only place a credential location is written, so a route can never point at a file of its own. +- **Binding metadata is not secret.** IDs, URLs, and timing values appear in logs and configuration dumps. Credential values never do, in logs, error messages, or metric labels. + +## Operations + +**Rollback.** Secret injection is confined to the `secrets` block and the header value. To go back to a static credential, replace the reference with a literal value and remove the block: + +```yaml +set_request_headers: + Authorization: 'Bearer token-v2' +``` + +Pomerium watches the mounted `config.yaml` and reloads it, so saving the file is enough. `docker compose up -d` will not recreate the container for a content change alone. To force a reload, restart the service: + +```bash +docker compose restart pomerium +``` + +A route that still references a binding you removed is a configuration error, so a partial rollback fails loudly and the old configuration stays in effect rather than failing at request time. + +**Change the tuning without downtime.** Editing `refresh` or `stale_grace` is picked up by the same reload. Values already in memory are kept, and Pomerium re-reads only bindings whose URL changed. + +**Cleanup.** Remove the stack and the credential: + +```bash +docker compose down -v +rm -rf secrets +``` + +`-v` also removes the `pomerium-cache` volume, which holds the Let's Encrypt certificate cache, so a rebuilt stack requests a new certificate. Then revoke the token at whatever issued it: deleting the file stops Pomerium from sending the credential, but does not make it invalid. + +## Next steps + +- [Secrets reference](/docs/reference/secrets) for every field, the failure matrix, and the JSON field selector for credentials stored in a JSON file +- [Headers Settings](/docs/reference/routes/headers#set-request-headers) for the other substitutions available in request headers +- [Metrics](/docs/reference/metrics#secrets) for the instruments to alert on diff --git a/content/docs/reference/metrics.mdx b/content/docs/reference/metrics.mdx index db6eddd07..b780e1586 100644 --- a/content/docs/reference/metrics.mdx +++ b/content/docs/reference/metrics.mdx @@ -113,6 +113,27 @@ Identity manager metrics have a `pomerium_identity_manager` prefix. | user_refresh_success | Counter | User refresh success counter. | | user_refresh_success_timestamp | Gauge | Timestamp of last successful user refresh. | +### Secrets \{#secrets} + +These metrics cover [secret injection](/docs/reference/secrets): reading credentials from a backend and substituting them into request headers. They have a `pomerium_secrets` prefix and are exported by the Authorize service. + +| Name | Type | Labels | Description | +| --- | --- | --- | --- | +| fetches_total | Counter | `provider`, `outcome` | Reads of a secret backend. `outcome` is `success`, `error`, or `not_found`. | +| fetch_duration | Histogram | `provider`, `outcome` | Duration of a read, in milliseconds. | +| refs_registered | Gauge | `provider` | Number of configured secret bindings. | +| cache_state | Gauge | `ref_label`, `state` | One series per binding, reporting its current state: `fresh`, `stale`, `expired`, or `failed`. | +| serving_stale_total | Counter | `ref_label` | Failed re-reads that left a binding serving its last good value. Counts read attempts, not requests. | +| negative_cache_hits_total | Counter | `ref_label` | Read attempts skipped because the backend recently reported the secret as missing. | +| singleflight_collapsed_total | Counter | `provider` | Concurrent reads of the same backend collapsed into one. | +| header_inject_total | Counter | `route_id`, `outcome` | Header injection attempts, counted once per header that carries a secret reference. `outcome` is `injected` or `rejected`. | + +`ref_label` is the binding ID. + +Header injection runs during authorization, before the allow or deny decision is applied, so `header_inject_total` also counts requests the policy then denies or redirects to sign-in. A `rejected` outcome means the header could not be built; the request is answered with `503 Service Unavailable` only if the policy allowed it. + +For alerting, `cache_state` is the clearest signal: `state="expired"` means a value was dropped after its grace window, and `state="failed"` means a binding has never been read successfully. `header_inject_total{outcome="rejected"}` shows the request-level effect, including values that were read but cannot be used as a header. + #### Envoy Proxy Metrics As of `v0.9`, Pomerium uses [Envoy](https://www.envoyproxy.io/) for the data plane. As such, proxy related metrics are sourced from Envoy, and use Envoy's internal [stats data model](https://www.envoyproxy.io/docs/envoy/latest/operations/stats_overview). Please see Envoy's documentation for information about specific metrics. diff --git a/content/docs/reference/reference.json b/content/docs/reference/reference.json index 60ecc4dde..fa65d6c54 100644 --- a/content/docs/reference/reference.json +++ b/content/docs/reference/reference.json @@ -1598,6 +1598,59 @@ "title": "Secondary Color (Dark Mode)", "type": "string" }, + "secrets": { + "description": "Read credentials from a file outside the Pomerium configuration and inject them into upstream request headers, with live rotation and fail-closed behavior.", + "id": "secrets", + "path": "/secrets", + "services": ["authorize"], + "short_description": "", + "title": "Secrets" + }, + "secrets-binding-refresh": { + "description": "How often Pomerium re-reads a secret binding's backend. Minimum one second.", + "id": "secrets-binding-refresh", + "path": "/secrets#bindings", + "services": ["authorize"], + "short_description": "", + "title": "Secrets Binding Refresh", + "type": "duration" + }, + "secrets-binding-stale-grace": { + "description": "How long a secret binding's last good value is still used while re-reading keeps failing. After this window the value is dropped and requests that need it are rejected with 503.", + "id": "secrets-binding-stale-grace", + "path": "/secrets#bindings", + "services": ["authorize"], + "short_description": "", + "title": "Secrets Binding Stale Grace", + "type": "duration" + }, + "secrets-binding-url": { + "description": "The backend a secret binding reads from. Supports file:// URLs with an absolute path, and an optional dotted-path fragment selecting one field of a JSON payload.", + "id": "secrets-binding-url", + "path": "/secrets#binding-urls", + "services": ["authorize"], + "short_description": "", + "title": "Secrets Binding URL", + "type": "string" + }, + "secrets-bindings": { + "description": "The secret binding table. Each key is a binding ID referenced from a route as ${secret.ID}, and each value describes the backend URL and how often to re-read it.", + "id": "secrets-bindings", + "path": "/secrets#bindings", + "services": ["authorize"], + "short_description": "", + "title": "Secrets Bindings", + "type": "map of objects" + }, + "secrets-defaults": { + "description": "Tuning values applied to secret bindings that leave a field unset: refresh, stale_grace, and negative_ttl.", + "id": "secrets-defaults", + "path": "/secrets#defaults", + "services": ["authorize"], + "short_description": "", + "title": "Secrets Defaults", + "type": "object" + }, "service-account-description": { "description": "A customizable description that identifies this service account.", "id": "service-account-description", diff --git a/content/docs/reference/routes/headers.mdx b/content/docs/reference/routes/headers.mdx index 99737712c..b056c9d89 100644 --- a/content/docs/reference/routes/headers.mdx +++ b/content/docs/reference/routes/headers.mdx @@ -138,7 +138,7 @@ remove_request_headers: The dynamic values enable you to pass ID and Access tokens from your identity provider to upstream applications. -To pass dynamic values from the user's OIDC claim to an upstream service, see [JWT Claim Headers](../jwt-claim-headers). +To pass dynamic values from the user's OIDC claim to an upstream service, see [JWT Claim Headers](../jwt-claim-headers). To inject a credential that Pomerium reads from a file outside its own configuration, see [Inject Secrets](#inject-secrets). :::caution @@ -240,6 +240,32 @@ Be very careful when passing access tokens to an upstream application. This may ::: +### Inject Secrets \{#inject-secrets} + +A header value can also carry a `${secret.ID}` reference. `ID` names an entry in the [`secrets.bindings`](/docs/reference/secrets#bindings) table, which tells Pomerium where to read the credential from: + +```yaml +secrets: + bindings: + upstream-api-token: + url: 'file:///etc/pomerium/secrets/upstream-api-token' + +routes: + - from: https://api.corp.example.com + to: https://api.internal + set_request_headers: + Authorization: 'Bearer ${secret.upstream-api-token}' + policy: + - allow: + or: + - email: + is: user@example.com +``` + +Unlike a static header value, the credential is not part of your Pomerium configuration. Pomerium re-reads the file in the background, so you can rotate the credential without a configuration change or a restart. If the credential cannot be read, Pomerium rejects the request with `503 Service Unavailable` instead of sending it without the header. + +Secret references work only in `set_request_headers`. For the full behavior, including rotation, failure handling, and monitoring, see the [Secrets](/docs/reference/secrets) reference. + ### Rewrite Request Headers \{#rewrite-request-headers} In addition to token substitutions, request headers can be rewritten using `${pomerium.request.headers["HEADER-NAME"]}`. For example if a request had a header `X-Jwt: JWT` and you would like to send it as `Authorization: Bearer JWT`, you could do so using: @@ -306,6 +332,8 @@ ingress.pomerium.io/remove_request_headers: | **Set Response Headers** allows you to set static values for the given response headers. These headers will take precedence over the global [`set_response_headers`](/docs/reference/set-response-headers). +Response header values are static text. `${pomerium.*}` and [`${secret.ID}`](/docs/reference/secrets) references are not substituted here, and a secret reference in a response header is a configuration error. + ### How to Configure \{#how-to-configure-set-response-headers} diff --git a/content/docs/reference/secrets.mdx b/content/docs/reference/secrets.mdx new file mode 100644 index 000000000..29487da8b --- /dev/null +++ b/content/docs/reference/secrets.mdx @@ -0,0 +1,296 @@ +--- +id: secrets +title: Secrets +sidebar_label: Secrets +description: Secrets settings let Pomerium read credentials from an external store and inject them into upstream request headers, with live rotation and fail-closed behavior. +keywords: + - reference + - secrets + - secret injection + - credentials + - rotation +pagination_prev: null +pagination_next: null +toc_max_heading_level: 2 +--- + +# Secrets + +**Secrets** settings let a route inject a credential that Pomerium reads from outside its own configuration. You give the credential a short name, called a **binding**, and point that binding at a file. A route then references the binding by name in [Set Request Headers](/docs/reference/routes/headers#set-request-headers): + +```yaml +secrets: + bindings: + upstream-api-token: + url: 'file:///etc/pomerium/secrets/upstream-api-token' + +routes: + - from: https://app.corp.example.com + to: https://api.internal + set_request_headers: + Authorization: 'Bearer ${secret.upstream-api-token}' + policy: + - allow: + or: + - email: + is: user@example.com +``` + +Pomerium reads the file in the background and substitutes the value on every request. You can rotate the credential by writing a new value to the file: Pomerium picks it up within seconds, with no configuration change and no restart. + +These settings are grouped in the configuration file under the key `secrets`. + +## Why use a binding \{#why-use-a-binding} + +Writing a credential directly into `set_request_headers` works, but the credential then lives in your Pomerium configuration. Every rotation is a configuration change, and the value is visible wherever the configuration is visible. + +With a binding: + +- The credential stays in a file that some other system owns and rotates. Your Pomerium configuration holds only the binding name and the file path. +- Rotation needs no configuration reload. +- If the credential cannot be read, Pomerium rejects the request instead of sending it without the header. See [Failure behavior](#failure-behavior). + +## Secret references \{#secret-references} + +A secret reference has the form `${secret.ID}`, where `ID` is a key in the binding table. References work only in a route's [`set_request_headers`](/docs/reference/routes/headers#set-request-headers). + +Rules: + +- A reference may sit next to literal text and next to the `${pomerium.*}` references listed in [Pass Dynamic Tokens in Headers](/docs/reference/routes/headers#pass-dynamic-tokens-in-headers), for example `'Bearer ${secret.upstream-api-token}'` or `'${pomerium.request.headers["X-Tenant"]}:${secret.upstream-api-token}'`. +- The same binding may be referenced many times, in many headers and many routes. Pomerium reads the backend once and shares the value. +- A reference takes exactly one name segment. `${secret.a.b}` is a configuration error. +- The name must be static. You cannot build it from request data, so `${secret.${pomerium.email}}` is a configuration error. +- To write a literal `$` in a header value, use `$$`. The value `$$secret.tok` is sent as the text `$secret.tok`. + +## Bindings \{#bindings} + +`secrets.bindings` is the binding table. Each key is a binding ID, and each value describes where the credential comes from and how often to re-read it. + +```yaml +secrets: + bindings: + upstream-api-token: + url: 'file:///etc/pomerium/secrets/upstream-api-token' + refresh: 1m + stale_grace: 10m +``` + +A binding ID must match `^[a-zA-Z0-9_][a-zA-Z0-9_-]*$`. Letters, digits, underscores, and dashes are allowed, the first character cannot be a dash, and dots are not allowed (a dot would be ambiguous inside `${secret.a.b}`). + +| Field | Type | Usage | Default | Meaning | +| :-- | :-- | :-- | :-- | :-- | +| `url` | string | **required** | — | Where the credential comes from. See [Binding URLs](#binding-urls). | +| `refresh` | duration | **optional** | `secrets.defaults` | How often Pomerium re-reads the backend. Minimum `1s`. | +| `stale_grace` | duration | **optional** | `secrets.defaults` | How long the last good value is still used while re-reading keeps failing, measured from the last successful read. | + +Bindings that point at the same URL share one read loop, so the smallest `refresh` among them sets the read interval for all of them. Bindings that point at the same URL **and** select the same field also share one cached value, and the smallest `stale_grace` applies. + +## Binding URLs \{#binding-urls} + +The `url` field takes a `file://` URL that points to a file Pomerium can read: + +```yaml +url: 'file:///etc/pomerium/secrets/upstream-api-token' +``` + +The URL must have no host component and an absolute path, which is what the three slashes in `file:///etc/...` produce. A form like `file://etc/pomerium/token` is a configuration error, because `etc` is read as a host name. Query parameters are not accepted, and the URL cannot contain `${...}`. + +The file content is the credential, byte for byte, with one exception: Pomerium removes exactly one trailing newline (`\n` or `\r\n`). Text editors and shell redirection usually add one. Any other whitespace is kept. + +### Selecting a field from a JSON file \{#json-selector} + +If the file holds a JSON document, add a fragment to the URL to select one field. The fragment is a dotted path: + +```yaml +secrets: + bindings: + api-key: + url: 'file:///etc/pomerium/secrets/credentials.json#data.token' +``` + +With the file below, `${secret.api-key}` is `s3cr3t`: + +```json +{"data": {"token": "s3cr3t", "expires": "2026-01-01"}} +``` + +Notes: + +- The selected value must be a string, a number, or a boolean. An object, an array, or `null` cannot be a header value. +- A value of the wrong type, a path that does not exist, and a file that is not valid JSON all count as failures for that binding. See [Failure behavior](#failure-behavior). The read itself still counts as a success, so `pomerium_secrets_fetches_total` reports `outcome="success"` while the binding's state is `failed` or `stale`. +- Two bindings may select different fields of the same file. Pomerium reads the file once, applies both selectors, and a failure in one does not affect the other. + +## Defaults \{#defaults} + +`secrets.defaults` sets the values used by bindings that leave a field unset. + +```yaml +secrets: + defaults: + refresh: 5m + stale_grace: 30m + negative_ttl: 30s +``` + +| Field | Type | Usage | Default | Meaning | +| :-- | :-- | :-- | :-- | :-- | +| `refresh` | duration | **optional** | `5m` | Default re-read interval. Minimum `1s`. | +| `stale_grace` | duration | **optional** | `30m` | Default grace window for serving the last good value while re-reading fails. | +| `negative_ttl` | duration | **optional** | `30s` | How long Pomerium waits before it reads again after the backend reports the secret as missing (a deleted file). This also delays recovery: a restored file is picked up at the end of this window, not immediately. | + +`negative_ttl` can only be set here, not per binding. + +## How to configure \{#how-to-configure} + +| **Config file key** | **Environment variable** | **Type** | **Usage** | +| :-- | :-- | :-- | :-- | +| `secrets.bindings` | `SECRETS_BINDINGS` | map of objects | **optional** | +| `secrets.defaults.refresh` | `SECRETS_DEFAULTS_REFRESH` | duration | **optional** | +| `secrets.defaults.stale_grace` | `SECRETS_DEFAULTS_STALE_GRACE` | duration | **optional** | +| `secrets.defaults.negative_ttl` | `SECRETS_DEFAULTS_NEGATIVE_TTL` | duration | **optional** | + +Write the binding table in the configuration file. The `SECRETS_BINDINGS` environment variable does accept the whole table as a JSON object, but that form is easy to get wrong and fails quietly: + +- Durations must be integers in nanoseconds, so `refresh: 1m` becomes `60000000000`. A duration string is a hard error. +- Keys are matched against Go field names, so `stale_grace` must be written `staleGrace`. +- Any key that matches nothing is ignored without a warning. + +### Example \{#example} + +```yaml +secrets: + defaults: + refresh: 1m + stale_grace: 10m + bindings: + upstream-api-token: + url: 'file:///etc/pomerium/secrets/upstream-api-token' + grafana-admin-password: + url: 'file:///etc/pomerium/secrets/grafana.json#admin.password' + refresh: 5m + +routes: + - from: https://api.corp.example.com + to: https://api.internal + set_request_headers: + Authorization: 'Bearer ${secret.upstream-api-token}' + policy: + - allow: + or: + - domain: + is: example.com + + - from: https://grafana.corp.example.com + to: http://grafana.internal:3000 + set_request_headers: + X-Grafana-Password: '${secret.grafana-admin-password}' + policy: + - allow: + or: + - domain: + is: example.com +``` + +## How Pomerium reads secrets \{#how-pomerium-reads-secrets} + +The Authorize service owns this work, because it is the service that builds request headers. In a [split service deployment](/docs/internals/configuration#all-in-one-vs-split-service-mode), the file must exist on the Authorize hosts, and the Authorize service must have permission to read it. No other service reads it. + +The `secrets` block itself is a different matter: every service that loads the routes also validates them, so a service whose configuration carries a route with `${secret.…}` but no matching binding refuses to start. Keep the block in the configuration of each service, and the credential file only where Authorize runs. + +Pomerium reads each backend in the background: + +- Reading starts as soon as the configuration is loaded, and does not block startup. A request that arrives before the first successful read is rejected. See [Failure behavior](#failure-behavior). +- Pomerium re-reads the file every `refresh` interval, and also watches the file for changes, so a rotation is usually visible in about a second. This includes an atomic replacement, such as a Kubernetes `Secret` volume update, where the file is swapped through a symbolic link. +- Requests never wait for a read. Substitution uses the value Pomerium already holds in memory. +- All references in one request read the same point-in-time value. Two requests taken during a rotation may see different values. +- Changing the configuration does not discard values that are already in memory. Pomerium only re-reads a binding whose URL changed, and stops reading bindings you removed. + +A validated configuration does not prove the file exists. Pomerium checks the shape of the URL when it loads the configuration, and reads the file afterwards. + +## Failure behavior \{#failure-behavior} + +Secret injection fails closed. If a header needs a secret that Pomerium cannot supply, the request is rejected with `503 Service Unavailable`. Pomerium never sends the request with the header missing, and never sends an empty value. + +| Situation | Behavior | +| :-- | :-- | +| Value read successfully | Header is injected. | +| Re-read failing, last good value within `stale_grace` | Header is injected with the last good value. Pomerium keeps retrying and logs a warning. | +| Re-read failing for longer than `stale_grace` | Value is dropped from memory. Requests that reference it get `503`. | +| No successful read yet, including a missing file at startup | Requests that reference the binding get `503`. | +| Value contains a carriage return, a line feed, or a null byte | Rejected when the header is built, so requests that reference it get `503`. The cached value still reads as `fresh`, because the read succeeded. | + +The `stale_grace` window runs from the last successful read, not from the first failure, and the value is dropped on the first failed re-read after the window ends. Both effects shift the moment requests start failing: with `refresh: 1m` and `stale_grace: 5m`, a file deleted right after a successful read is served for about five and a half more minutes, while one deleted just before the next read is served for about four and a half. Pomerium retries a missing secret every `negative_ttl`, and other read errors on a backoff that grows to at most 30 seconds. + +Only routes that reference an affected binding are rejected. Other routes are unaffected, including routes on the same Pomerium instance that reference healthy bindings. + +A policy denial always wins over a secret failure, so an authorization decision is never reported as `503`. + +A browser gets Pomerium's standard error page, which shows `503 Service Unavailable` and no detail about the secret. A client that asks for JSON, and a Kubernetes client, receive the reason `secret unavailable` in the response body. The binding and the header are named only in the Authorize service log: + +```json +{ + "level": "warn", + "binding": "upstream-api-token", + "header": "authorization", + "route": "https://app.corp.example.com → https://api.internal", + "message": "authorize: secret unavailable, denying request" +} +``` + +## Monitoring \{#monitoring} + +Pomerium logs a secret's state changes rather than each request. These lines carry the backend URL, the binding ID, the new state, and, after a failure, an error class. The last line in the table below is the exception: it identifies the secret by URL only, with no binding ID and no state. None of them carry the value. + +| Level | Message | Meaning | +| :-- | :-- | :-- | +| `info` | `secret resolved` | First successful read. | +| `info` | `secret recovered` | A successful read after a failure. | +| `warn` | `secret serving stale` | Re-reading is failing, and the last good value is still being used. | +| `error` | `secret expired` | The grace window elapsed, and the value was dropped. | +| `warn` | `secret not found; negative-caching` | The backend reports the secret is missing, for example a deleted file. This line names the URL, not the binding, and repeats at most once per `negative_ttl`. | + +Because these lines mark state changes, a binding that never reads successfully, for example a file Pomerium is not allowed to read or a selector that never matches, produces no line of its own: its state never changes. The metric `pomerium_secrets_cache_state{state="failed"}` is the reliable signal for that case. + +Two more lines are written per rejected request, so they can be frequent: + +| Level | Message | Fields | +| :-- | :-- | :-- | +| `warn` | `authorize/header-evaluator: secret unavailable, rejecting request` | `route_id`, `error_class` (`unavailable` or `invalid_value`) | +| `warn` | `authorize: secret unavailable, denying request` | `binding`, `header`, `route` | + +The first is written whenever a header cannot be built, which includes requests the policy then denies for its own reasons. The second is written only when the request would otherwise have been allowed, and is therefore the one that corresponds to a `503`. + +Metrics for reads, cache state, and injection outcomes are described in [Metrics](/docs/reference/metrics#secrets). + +## Configuration errors \{#configuration-errors} + +Secret problems that can be found before serving traffic are configuration errors. Pomerium refuses to start, and on a configuration reload it logs `config: error updating config` and keeps the previous configuration. + +| Message | Cause | +| :-- | :-- | +| `unknown secret "typo"` | A header references an ID that is not in the binding table. | +| `routes reference secrets [...] but no secrets bindings are configured` | Headers use `${secret.…}` and there is no `secrets.bindings` block. | +| `secret refs take exactly one ID segment` | A reference such as `${secret.a.b}`. | +| `malformed secret reference` | A reference built from other references, such as `${secret.${pomerium.email}}`. | +| `secret refs are not supported in response headers in v1` | `${secret.…}` used in `set_response_headers`, per route or globally. | +| `invalid ID, must match ^[a-zA-Z0-9_][a-zA-Z0-9_-]*$` | A binding ID with a dot, a leading dash, or another disallowed character. | +| `refresh 100ms below minimum 1s` | `refresh` below the one-second floor. | +| `unknown scheme "vault" (known schemes: [file])` | A binding URL with a scheme Pomerium does not support. | +| `file URL must not have a host component` | A `file://` URL with a relative path, such as `file://etc/token`. | +| `URL must not contain "${...}" interpolation` | A binding URL built with a template reference. | +| `file secret: unsupported query parameters` | A `file://` URL with query parameters. | + +These checks cover the routes in Pomerium's own configuration. Routes delivered by a management plane, such as the Kubernetes Ingress controller, Pomerium Zero, or Pomerium Enterprise, are added after this validation runs, so a bad reference in one of those routes is not caught here. It fails closed at request time instead. + +## Security notes \{#security-notes} + +- Anyone who can edit a route can decide which binding that route injects, but not where a credential comes from. Only the binding table names backend locations. +- Binding IDs, URLs, and timing values are ordinary configuration. They appear in logs and configuration dumps. Credential values do not. +- A secret is only as protected as the file holding it. Restrict the file to the user the Authorize service runs as, and keep it out of images and version control. +- Be careful about which upstream receives a credential. An upstream that receives a shared token can use it for anything that token allows. + +## Limitations \{#limitations} + +- `file://` is the only supported backend. +- References work only in a route's `set_request_headers`. Response header values are never substituted, and a reference in a response header in the configuration file is rejected. There is no global request header setting. +- The `secrets` block is set in the Pomerium configuration file or through environment variables. There is no Ingress annotation for it. diff --git a/content/examples/guides/secrets/config.yaml.md b/content/examples/guides/secrets/config.yaml.md new file mode 100644 index 000000000..a2991a3c0 --- /dev/null +++ b/content/examples/guides/secrets/config.yaml.md @@ -0,0 +1,40 @@ +```yaml title="config.yaml" +# Pomerium Core configuration. Uses the hosted authenticate service, so you +# don't run your own identity provider. To self-host the IdP, see the OIDC +# guide: https://www.pomerium.com/docs/integrations/user-identity/oidc +authenticate_service_url: https://authenticate.pomerium.app + +# Obtain TLS certificates automatically from Let's Encrypt. +autocert: true + +# Prometheus metrics, published to localhost only by the Compose file. +metrics_address: :9090 + +# The binding table names every credential Pomerium may inject, and says where +# each one comes from. Routes reference a binding by ID and never name a path, +# so this block is the only place a credential location is written down. +secrets: + defaults: + # Re-read every binding at least this often. Pomerium also watches the file, + # so a rotation is usually visible in about a second. + refresh: 1m + # If re-reading fails, keep using the last good value for this long. After + # that, requests that need the credential are rejected with 503. + stale_grace: 5m + bindings: + upstream-api-token: + url: 'file:///etc/pomerium/secrets/upstream-api-token' + +routes: + - from: https://api.yourdomain.com + to: http://upstream:8080 + # Pomerium substitutes the current value of the binding on every request. + # The credential itself is not part of this file. + set_request_headers: + Authorization: 'Bearer ${secret.upstream-api-token}' + policy: + - allow: + or: + - email: + is: you@example.com +``` diff --git a/content/examples/guides/secrets/docker-compose.yaml.md b/content/examples/guides/secrets/docker-compose.yaml.md new file mode 100644 index 000000000..39539b689 --- /dev/null +++ b/content/examples/guides/secrets/docker-compose.yaml.md @@ -0,0 +1,42 @@ +```yaml title="docker-compose.yaml" +services: + pomerium: + image: pomerium/pomerium:main + volumes: + - ./config.yaml:/pomerium/config.yaml:ro + # Mount the directory, not the single file, so the credential can be + # replaced atomically from the host. Read-only: Pomerium never writes it. + - ./secrets:/etc/pomerium/secrets:ro + - pomerium-cache:/data + ports: + - 443:443 + - 80:80 + # Metrics stay on the loopback interface. + - 127.0.0.1:9090:9090 + # Pomerium bridges both networks: `default` for autocert/Let's Encrypt and the + # hosted authenticate service, and the internal-only network to reach the API. + networks: + - default + - api-internal + restart: always + + # Stand-in for the real upstream API. It echoes the request it received, + # including headers, so you can see what Pomerium injected. + upstream: + image: mendhak/http-https-echo:37 + environment: + HTTP_PORT: 8080 + # Internal-only network with no published ports: the API is reachable only + # through Pomerium, so a leaked credential cannot be replayed against it + # from outside. + networks: + - api-internal + restart: always + +networks: + api-internal: + internal: true + +volumes: + pomerium-cache: +``` diff --git a/cspell.json b/cspell.json index 695286f5d..21d25f2c9 100644 --- a/cspell.json +++ b/cspell.json @@ -19,6 +19,8 @@ "language": "en-US", "version": "0.2", "words": [ + "distroless", + "singleflight", "Ghostty", "yourdomain", "ABAC", From f2594b411a2eec54ba940b7d78c6c88c9dd2a875 Mon Sep 17 00:00:00 2001 From: Denis Mishin Date: Fri, 31 Jul 2026 14:44:29 -0400 Subject: [PATCH 2/2] docs: opt the secrets guide out of fixture validation, warn about older images --- content/docs/guides/secrets.mdx | 7 +++++++ content/examples/guides/secrets/validate/SKIP | 1 + content/examples/guides/secrets/validate/screenshots-skip | 1 + 3 files changed, 9 insertions(+) create mode 100644 content/examples/guides/secrets/validate/SKIP create mode 100644 content/examples/guides/secrets/validate/screenshots-skip diff --git a/content/docs/guides/secrets.mdx b/content/docs/guides/secrets.mdx index 8044f25d0..9c589511f 100644 --- a/content/docs/guides/secrets.mdx +++ b/content/docs/guides/secrets.mdx @@ -74,6 +74,12 @@ You also need: The Compose file uses the `pomerium/pomerium:main` image tag, because secret injection is newer than the latest pinned release. Pin a released tag once one includes it. +:::warning + +An image that predates the feature does not reject this configuration. It logs `{"level":"error","key":"secrets","message":"unknown config option"}`, starts anyway, and expands `${secret.upstream-api-token}` to an empty string, so the upstream receives `Authorization: Bearer ` with no credential. If your upstream returns `401` and Pomerium never logs `secret resolved`, check that log line first: you are running an image without secret injection. + +::: + ## Step 1: create the credential file Make a directory for credentials and write the token into it. Use `printf`, not `echo -n`, for predictable output: @@ -241,6 +247,7 @@ Recovery takes up to `negative_ttl`, 30 seconds by default, because Pomerium sto | Requests start failing right after a rotation, and the log says `secret not found` | The single file was bind-mounted instead of the directory, so replacing it left the container pointing at the old inode | The `volumes` entry in `docker-compose.yaml` | Mount the directory (`./secrets:/etc/pomerium/secrets:ro`) and recreate the container | | The upstream returns `401` while Pomerium injects the header | The value carries characters you did not intend | `xxd secrets/upstream-api-token \| tail -2`. Pomerium strips one trailing newline, and nothing else | Rewrite the file with `printf '%s' "$TOKEN"`. A value containing a carriage return, a line feed, or a null byte is rejected when the header is built, so those requests get `503` instead | | Only some routes return `503` | Expected. Rejection is per binding | Which bindings the failing routes reference | Fix the affected binding. Routes that reference healthy bindings are unaffected | +| The upstream returns `401`, and no `secret` log line ever appears | The image predates secret injection, so the reference expanded to an empty string | `docker compose logs pomerium \| grep 'unknown config option'` | Use an image that includes the feature. Without it there is no fail-closed behavior: the header is sent empty | The Pomerium image is distroless and has no shell, so `docker compose exec pomerium ls` will not work. Inspect the host directory and the volume definition instead. diff --git a/content/examples/guides/secrets/validate/SKIP b/content/examples/guides/secrets/validate/SKIP new file mode 100644 index 000000000..ce1200eae --- /dev/null +++ b/content/examples/guides/secrets/validate/SKIP @@ -0,0 +1 @@ +Not sealable yet: the fixture harness pins a released Pomerium image, and the `secrets` binding table it needs exists in no published release nor in `pomerium/pomerium:main`. An image without the feature does not fail loudly either: it logs `unknown config option` for `secrets`, starts, and expands `${secret.ID}` to an empty string, so a sealed run would assert against a silently empty header. The guide was instead verified manually against a Pomerium built from the secret-injection branch, in a Compose stack matching the one shipped here: injection into `set_request_headers`, live rotation by rewriting and by atomically replacing the file, the fail-closed `503` after `stale_grace`, recovery, and the `pomerium_secrets_*` metrics. Replace this marker with a real `compose.validate.yaml` once a published image includes secret injection. diff --git a/content/examples/guides/secrets/validate/screenshots-skip b/content/examples/guides/secrets/validate/screenshots-skip new file mode 100644 index 000000000..fea477ad5 --- /dev/null +++ b/content/examples/guides/secrets/validate/screenshots-skip @@ -0,0 +1 @@ +No app UI to capture: the upstream in this guide is an HTTP echo service, so verification is a JSON response body and log lines, both of which the guide quotes verbatim as text. A screenshot of that JSON would carry no information the quoted output does not.