diff --git a/content/docs/capabilities/non-http/tcp.mdx b/content/docs/capabilities/non-http/tcp.mdx index 87f5de96d..a66208644 100644 --- a/content/docs/capabilities/non-http/tcp.mdx +++ b/content/docs/capabilities/non-http/tcp.mdx @@ -150,6 +150,55 @@ routes: to: http://second-proxy.example.corp.com:10003 ``` +### Connect through a client-side forward proxy {#forward-proxy} + +When the machine running `pomerium-cli` can only reach the internet through an HTTP or SOCKS5 forward proxy, route the TCP tunnel through it with `--forward-proxy`: + +```bash +pomerium-cli tcp db.internal:5432 \ + --pomerium-url https://pomerium.example.com \ + --forward-proxy http://proxy.internal:3128 +``` + +The proxy carries the connection from `pomerium-cli` to the **Pomerium edge**, not to the upstream TCP service. Write proxy allowlists and `NO_PROXY` entries for the Pomerium edge hostname, not for the database, SSH server, or other upstream. This is different from [Proxy chaining support](#proxy-chaining-support), which places a second proxy behind Pomerium. + +:::info Requires pomerium-cli vX.Y or later + +Proxy-aware tunneling was added to `pomerium-cli` in vX.Y. Check your client with `pomerium-cli tcp --help | grep forward-proxy` and upgrade if `--forward-proxy` is not listed. + +::: + +`pomerium-cli` selects a proxy in this order: + +| Order | Source | `NO_PROXY` | Notes | +| --- | --- | --- | --- | +| 1 | `--forward-proxy` | Ignored | Authoritative; used as-is when set. | +| 2 | `HTTPS_PROXY` / `HTTP_PROXY` | Honored | Scheme-aware, based on the Pomerium edge URL. Lowercase variants also work. | +| 3 | `ALL_PROXY` | Honored | Fallback when no scheme-specific proxy applies. | +| 4 | none | n/a | Direct connection; unchanged from earlier behavior. | + +Both the tunnel and `pomerium-cli`'s own authentication and token requests use the selected proxy. If the command opens a system browser to log in, the browser uses its own network and proxy settings, not `--forward-proxy`. + +The proxy URL takes any of these forms (and must not include a path, query, or fragment): + +| Form | Meaning | +| --- | --- | +| `proxy.internal:3128` | Same as `http://proxy.internal:3128`. | +| `http://proxy.internal:3128` | HTTP CONNECT proxy. | +| `https://proxy.internal:8443` | TLS to the proxy, then HTTP CONNECT. | +| `socks5://proxy.internal:1080` | SOCKS5 proxy. | +| `socks5h://proxy.internal:1080` | SOCKS5 proxy (`socks5h` scheme). | +| `http://user:pass@proxy.internal:3128` | HTTP proxy with Basic authentication; URL-encode special characters. | + +A few more things to know: + +- TCP tunnels use the proxy; `pomerium-cli udp` (MASQUE) always dials directly. +- When a proxy is selected, `pomerium-cli tcp` skips HTTP/3 and uses the HTTP/1 tunnel, since QUIC can't traverse a CONNECT or SOCKS proxy. +- `--alternate-ca-path` configures trust for the Pomerium edge certificate. It does not configure trust for an `https://` forward proxy; a proxy that terminates TLS with a private CA must be trusted by the operating system running `pomerium-cli`. +- Credentials in an `http://user:pass@proxy` URL are sent to the proxy in cleartext. + +For a runnable walkthrough with Pomerium, an upstream service, and a Squid proxy, see [Connect through a client-side forward proxy](/docs/guides/tcp-forward-proxy). + :::info TCP examples The guides below demonstrate how to proxy TCP tunnels with Pomerium to well-known services: @@ -161,5 +210,6 @@ The guides below demonstrate how to proxy TCP tunnels with Pomerium to well-know - [**Redis**](./examples/redis) - [**SSH**](./examples/ssh) - [**SSH over port 443 through an L4 edge**](/docs/guides/ssh-tcp-l4-passthrough) - for environments restricted to outbound port 443 +- [**Connect through a client-side forward proxy**](/docs/guides/tcp-forward-proxy) - for clients whose only egress is an HTTP or SOCKS5 proxy ::: diff --git a/content/docs/guides/tcp-forward-proxy.mdx b/content/docs/guides/tcp-forward-proxy.mdx new file mode 100644 index 000000000..53a8407d9 --- /dev/null +++ b/content/docs/guides/tcp-forward-proxy.mdx @@ -0,0 +1,282 @@ +--- +title: 'Connect through a client-side forward proxy' +sidebar_label: 'TCP through a forward proxy' +lang: en-US +description: 'Tunnel a TCP route with pomerium-cli when the client can only reach the internet through an HTTP or SOCKS5 forward proxy.' +keywords: + [ + pomerium, + pomerium-cli, + tcp, + tcp routes, + forward proxy, + http connect, + socks5, + squid, + postgres, + egress proxy, + ] +--- + +import ComposeFile from '!!raw-loader!@site/content/examples/tcp-forward-proxy/docker-compose.yml'; +import SquidConf from '!!raw-loader!@site/content/examples/tcp-forward-proxy/squid/squid.conf'; + +import CodeBlock from '@theme/CodeBlock'; + +# Connect through a client-side forward proxy + +This guide shows how to use `pomerium-cli tcp` from a network whose only outbound path is an HTTP or SOCKS5 forward proxy, such as a corporate egress proxy. The `--forward-proxy` flag routes the tunnel through that proxy to reach Pomerium. The local example tunnels to a Postgres database and runs `psql` over the tunnel. + +This is about the client's egress path. It is different from [Proxy chaining support](/docs/capabilities/non-http/tcp#proxy-chaining-support), which puts a second proxy behind Pomerium. + +## How it works + +```mermaid +flowchart LR + A[psql] --> B[pomerium-cli tcp
local listener] + B --> C[Client-side forward proxy
Squid or SOCKS5] + C --> D[Pomerium edge
terminates TLS] + D --> E[Postgres] +``` + +The forward proxy connects `pomerium-cli` to the Pomerium edge, not to the upstream service. Write proxy allowlists and `NO_PROXY` entries for the Pomerium edge hostname, not for the database. For flag and environment-variable precedence and the supported proxy URL forms, see [Connect through a client-side forward proxy](/docs/capabilities/non-http/tcp#forward-proxy) in the TCP reference. + +:::info Requires pomerium-cli vX.Y or later + +Proxy-aware tunneling was added to `pomerium-cli` in vX.Y. Check your client with `pomerium-cli tcp --help | grep forward-proxy` and upgrade if `--forward-proxy` is not listed. + +::: + +## Prerequisites + +- Docker Compose v2 +- OpenSSL +- Git + +## Run the local example + +The runnable example lives in [`content/examples/tcp-forward-proxy`](https://github.com/pomerium/documentation/tree/main/content/examples/tcp-forward-proxy). It runs Postgres, Pomerium, a [Squid](http://www.squid-cache.org/) HTTP proxy, a SOCKS5 proxy, and a client container with `pomerium-cli` and `psql`. The client reaches a proxy by name, and the proxy reaches the Pomerium edge by its Docker network alias. + +1. Clone the docs repository and enter the example directory. + + ```bash + git clone https://github.com/pomerium/documentation + cd documentation/content/examples/tcp-forward-proxy + ``` + +1. Generate a self-signed certificate for the demo hostnames. + + ```bash + ./gen-certs.sh + ``` + + The script writes a self-signed certificate for `*.localhost.pomerium.io` that `pomerium-cli` trusts with `--alternate-ca-path`. In production, use a publicly trusted certificate or your organization's managed trust chain. + +1. Review the routes in `config/pomerium.yaml`. One public route demonstrates the tunnel; one policy-gated route demonstrates enforcement. + + ```yaml + routes: + - from: tcp+https://pgsql.localhost.pomerium.io:5432 + to: tcp://postgres:5432 + allow_public_unauthenticated_access: true + - from: tcp+https://secure.localhost.pomerium.io:5432 + to: tcp://postgres:5432 + policy: + - allow: + and: + - authenticated_user: true + ``` + + :::caution Lab-only public route + + The first route is public so the lab needs no identity provider. Do not use `allow_public_unauthenticated_access` for production TCP services. See [Require authentication in production](#require-authentication-in-production). + + ::: + +1. Start the stack. + + ```bash + docker compose up -d --build + docker compose ps + ``` + +1. Open the tunnel through the HTTP proxy. + + ```bash + docker compose exec -d client sh -lc \ + 'pomerium-cli tcp pgsql.localhost.pomerium.io:5432 \ + --pomerium-url https://proxy.localhost.pomerium.io \ + --alternate-ca-path /certs/pomerium.crt \ + --forward-proxy http://squid:3128 \ + --browser-cmd /bin/true \ + --listen 127.0.0.1:5432 \ + >/tmp/pomerium-cli.log 2>&1' + ``` + + `--browser-cmd /bin/true` suppresses the browser launch in the headless client container; drop it for interactive use. The public lab route never triggers a login. + +1. Run a query through the tunnel. `PGPASSWORD` is fine for this throwaway lab; prefer `~/.pgpass` or a prompt elsewhere. + + ```bash + docker compose exec client sh -lc \ + 'PGPASSWORD=postgres psql -h 127.0.0.1 -p 5432 -U postgres </tmp/pomerium-cli-socks5.log 2>&1' + +docker compose exec client sh -lc \ + 'PGPASSWORD=postgres psql -h 127.0.0.1 -p 5433 -U postgres -c "select 1;"' +``` + +## Verify the path + +Confirm the tunnel traversed Squid. The CONNECT target is the Pomerium edge, not the upstream: + +```bash +docker compose exec squid grep CONNECT /var/log/squid/access.log +``` + +```text +... CONNECT proxy.localhost.pomerium.io:443 ... +``` + +Check Pomerium's access log for the tunnel: + +```bash +docker compose logs pomerium | grep '"method":"CONNECT"' +``` + +Look for the route-match host and a successful response: + +```json +"host":"pgsql.localhost.pomerium.io:5432" +"response-code":200 +``` + +For log field definitions, see [Access Log Fields](/docs/reference/access-log-fields). + +## Confirm Pomerium enforces access + +The proxy gets you to Pomerium, but Pomerium still decides who reaches the upstream. Open a tunnel to the policy-gated route with no credentials: + +```bash +docker compose exec -d client sh -lc \ + 'pomerium-cli tcp secure.localhost.pomerium.io:5432 \ + --pomerium-url https://proxy.localhost.pomerium.io \ + --alternate-ca-path /certs/pomerium.crt \ + --forward-proxy http://squid:3128 \ + --browser-cmd /bin/true \ + --listen 127.0.0.1:5499 \ + >/tmp/pomerium-cli-secure.log 2>&1' + +docker compose exec client sh -lc \ + 'PGCONNECT_TIMEOUT=8 PGPASSWORD=postgres psql -h 127.0.0.1 -p 5499 -U postgres -c "select 1;"' +``` + +The connection fails: Pomerium requires authentication before it will open the tunnel, so `psql` never reaches Postgres. The authorize log shows the denial: + +```bash +docker compose logs pomerium | grep secure.localhost.pomerium.io +``` + +```json +"host":"secure.localhost.pomerium.io:5432" +"allow":false +"allow-why-false":["user-unauthenticated"] +``` + +## Require authentication in production + +The first lab route is public to keep the example self-contained. In production, drop `allow_public_unauthenticated_access` and attach a policy, as the `secure` route above does: + +```yaml +routes: + - from: tcp+https://pgsql.example.com:5432 + to: tcp://postgres.internal:5432 + policy: + - allow: + and: + - email: + is: data-team@example.com +``` + +You also need an identity provider. To avoid running your own, point `authenticate_service_url` at Pomerium's hosted authenticate service (`https://authenticate.pomerium.app`), as the [Get Started guide](/docs/get-started/fundamentals/core/get-started) does, or use [Pomerium Zero](/docs/get-started/fundamentals/zero/zero-build-routes). The first time `pomerium-cli` opens a protected tunnel it runs the OIDC login flow, and that login uses the same `--forward-proxy` as the tunnel. For non-interactive access, [Pomerium Zero and Enterprise](/docs/capabilities/service-accounts) issue service-account tokens you pass with `--service-account-file`. + +## Tunnel SSH instead + +The same pattern works for SSH. Use `pomerium-cli` as an SSH `ProxyCommand` and add the proxy with `--forward-proxy` (or `HTTPS_PROXY`): + +```ssh-config +Host db-bastion + HostName ssh.example.com + Port 22 + ProxyCommand pomerium-cli tcp --listen - %h:%p --pomerium-url https://pomerium.example.com --forward-proxy http://proxy.internal:3128 +``` + +For SSH-specific networking, including running Pomerium behind an L4 edge, see [SSH over port 443 through an L4 edge](/docs/guides/ssh-tcp-l4-passthrough). + +## Troubleshoot + +| Symptom | Likely cause | Fix | +| --- | --- | --- | +| No `CONNECT` line in the Squid log | No proxy was selected, or `NO_PROXY` matched the edge host | Check `--forward-proxy`, `HTTPS_PROXY`, and `NO_PROXY`. | +| `proxy CONNECT failed: 407` | The proxy requires authentication | Add credentials to the proxy URL, URL-encoding special characters. | +| TLS error reaching Pomerium | The edge certificate is not trusted | Pass `--alternate-ca-path`, or install the CA. | +| TLS error reaching an `https://` proxy | The proxy certificate is not trusted by the OS | Install the proxy CA into the system trust store; `--alternate-ca-path` does not affect the proxy hop. | +| `pomerium-cli udp` ignores the proxy | Expected | This feature applies to `pomerium-cli tcp` only. | + +## Clean up + +```bash +docker compose down -v +rm -f certs/pomerium.crt certs/pomerium.key +``` + +## Reference + +The full Compose file and Squid config: + + + {ComposeFile} + + + + {SquidConf} + + +## More resources + +- [TCP support](/docs/capabilities/non-http/tcp) +- [PostgreSQL over TCP](/docs/capabilities/non-http/examples/postgres) +- [Pomerium Policy Language](/docs/internals/ppl) +- [Access Log Fields](/docs/reference/access-log-fields) diff --git a/content/examples/tcp-forward-proxy/.gitignore b/content/examples/tcp-forward-proxy/.gitignore new file mode 100644 index 000000000..6933bb784 --- /dev/null +++ b/content/examples/tcp-forward-proxy/.gitignore @@ -0,0 +1,2 @@ +certs/pomerium.crt +certs/pomerium.key diff --git a/content/examples/tcp-forward-proxy/README.md b/content/examples/tcp-forward-proxy/README.md new file mode 100644 index 000000000..9dd60b94a --- /dev/null +++ b/content/examples/tcp-forward-proxy/README.md @@ -0,0 +1,63 @@ +# pomerium-cli tcp through a client-side forward proxy + +This example supports the guide at: + +https://www.pomerium.com/docs/guides/tcp-forward-proxy + +It runs five services: + +- `postgres`: the upstream TCP service +- `pomerium`: the TCP route proxy / edge +- `squid`: an HTTP CONNECT forward proxy +- `socks5`: a SOCKS5 forward proxy +- `client`: in-network test client with `pomerium-cli` and `psql` + +The `client` image copies `pomerium-cli` from `pomerium/cli:latest`. The +`--forward-proxy` flag requires a `pomerium-cli` release that includes it; see +the guide for the minimum version. + +## Run + +Generate the local demo certificate first: + +```bash +./gen-certs.sh +``` + +Start the stack: + +```bash +docker compose up -d --build +docker compose ps +``` + +Tunnel to Postgres through the HTTP proxy and run a query: + +```bash +docker compose exec -d client sh -lc \ + 'pomerium-cli tcp pgsql.localhost.pomerium.io:5432 \ + --pomerium-url https://proxy.localhost.pomerium.io \ + --alternate-ca-path /certs/pomerium.crt \ + --forward-proxy http://squid:3128 \ + --browser-cmd /bin/true \ + --listen 127.0.0.1:5432 \ + >/tmp/pomerium-cli.log 2>&1' + +docker compose exec client sh -lc \ + 'PGPASSWORD=postgres psql -h 127.0.0.1 -p 5432 -U postgres -c "select version();"' +``` + +Through the SOCKS5 proxy instead, change the flag to `--forward-proxy socks5://socks5:1080`. + +Confirm the tunnel traversed the HTTP proxy: + +```bash +docker compose exec squid grep CONNECT /var/log/squid/access.log +``` + +Clean up: + +```bash +docker compose down -v +rm -f certs/pomerium.crt certs/pomerium.key +``` diff --git a/content/examples/tcp-forward-proxy/config/pomerium.yaml b/content/examples/tcp-forward-proxy/config/pomerium.yaml new file mode 100644 index 000000000..5ae25ab02 --- /dev/null +++ b/content/examples/tcp-forward-proxy/config/pomerium.yaml @@ -0,0 +1,33 @@ +address: :443 + +# Required by the config schema even though the public route below needs no +# login and no identity provider is configured. +authenticate_service_url: https://authenticate.localhost.pomerium.io + +certificate_file: /certs/pomerium.crt +certificate_key_file: /certs/pomerium.key + +# Demo-only static secrets. Use generated secrets in production, for example: +# head -c 32 /dev/urandom | base64 +shared_secret: SkqCcCo1LuOtr8pW6SyENjDadPH/j41HpLpXAnWtEKA= +cookie_secret: KTY/DF276AcHctxYwKU2RGBCq62KEDskPcEe3ImoMXQ= + +log_level: info + +routes: + # DEMO ONLY: public so the lab needs no identity provider. Replace + # allow_public_unauthenticated_access with a policy like the one below in + # production (see the guide). + - from: tcp+https://pgsql.localhost.pomerium.io:5432 + to: tcp://postgres:5432 + allow_public_unauthenticated_access: true + + # Same upstream, but policy-gated. With no identity provider configured, an + # unauthenticated tunnel to this route is denied, which shows Pomerium + # enforcing access before the TCP connection reaches Postgres. + - from: tcp+https://secure.localhost.pomerium.io:5432 + to: tcp://postgres:5432 + policy: + - allow: + and: + - authenticated_user: true diff --git a/content/examples/tcp-forward-proxy/docker-compose.yml b/content/examples/tcp-forward-proxy/docker-compose.yml new file mode 100644 index 000000000..423957a9a --- /dev/null +++ b/content/examples/tcp-forward-proxy/docker-compose.yml @@ -0,0 +1,82 @@ +name: pomerium-tcp-forward-proxy + +services: + # The upstream TCP service, reached with psql through the tunnel. + postgres: + image: postgres:16-alpine + environment: + POSTGRES_PASSWORD: postgres + healthcheck: + test: ['CMD-SHELL', 'pg_isready -U postgres'] + interval: 3s + timeout: 3s + retries: 20 + start_period: 5s + + pomerium: + image: pomerium/pomerium:latest + depends_on: + postgres: + condition: service_healthy + volumes: + - ./certs:/certs:ro + - ./config/pomerium.yaml:/pomerium.yaml:ro + command: ['--config', '/pomerium.yaml'] + networks: + default: + # The *.localhost.pomerium.io names resolve to 127.0.0.1 publicly, which + # is wrong inside Docker. These aliases let the proxies resolve the edge + # name to this container. + aliases: + - proxy.localhost.pomerium.io + - authenticate.localhost.pomerium.io + - pgsql.localhost.pomerium.io + healthcheck: + test: ['CMD', 'pomerium', 'health'] + interval: 3s + timeout: 3s + retries: 30 + start_period: 5s + + # An HTTP forward proxy standing in for a corporate egress proxy (HTTP CONNECT). + squid: + image: ubuntu/squid:latest + command: ['squid', '-N', '-d', '1', '-f', '/etc/squid/squid.conf'] + volumes: + - ./squid/squid.conf:/etc/squid/squid.conf:ro + healthcheck: + test: ['CMD-SHELL', 'bash -c "