feat: let the argocd and bao CLIs authenticate through oauth2-proxy

[venkatamutyala]

Problem

Neither the argocd nor the bao CLI can reach its backend on a GlueOps cluster. Verified on nonprod.jupiter.onglueops.rocks:

ArgoCD  /                               -> 302  oauth2/start
ArgoCD  /api/v1/session          (POST) -> 401  Unauthorized
ArgoCD  /version.VersionService/Version -> 302  oauth2/start
Vault   /v1/sys/seal-status             -> 302  oauth2/start

argocd version --grpc-web through the ingress returns only the client version; against a port-forward it returns argocd-server: v3.2.12. The backends are healthy — the edge is what stops them.

The existing argocd-server-api ingress (/api, oauth2-no-redirect) does not help. That chain still runs oauth2-forwardauth, so it turns the redirect into a bare 401; and the CLI does not use /api anyway — argocd-server logs show received unary call /version.VersionService/Version, a root-level gRPC path matched by the / ingress.

Three independent blockers, one per commit.

1. Dex had no client an operator could mint a token from

Add toolbox — public, so no client secret has to be distributed to developer machines. It is used with the OIDC device flow, which needs no redirect URI, so nothing is registered per developer and it works over SSH, in a container, or when the browser is on another machine entirely.

Deliberately not added: loopback callbacks on the argocd/vault clients. A localhost redirect only works when the browser and CLI share a host, which is the case this is meant to solve. Consumers pair this with ArgoCD's allowedAudiences (GlueOps/docs-argocd#63) and OpenBao's jwt-type roles (GlueOps/terraform-module-kubernetes-hashicorp-vault-configuration#68), so one token authenticates everywhere and no browser callback is involved.

2. oauth2-proxy only accepted a cookie

skip_jwt_bearer_tokens=true lets a Dex id_token satisfy the edge, with oidc_extra_audiences accepting the audience our CLIs present. Traefik already forwards the caller's Authorization header to the auth server (utils.CopyHeaders(forwardReq.Header, req.Header)), so nothing else is needed to get the token there.

3. forwardauth stripped the caller's own bearer token

oauth2-forwardauth lists authorization in authResponseHeaders. Traefik applies that list as:

req.Header.Del(headerKey)                        // unconditional
if len(forwardResponse.Header[headerKey]) > 0 {  // oauth2-proxy returns nothing

Since set_authorization_header is off, oauth2-proxy never returns that header, so the entry can only ever strip the caller's token before the backend sees it. oauth2-forwardauth-bearer is the same authentication without it, plus two chains:

• oauth2-api — pure API/gRPC routes; 401 rather than a login redirect
• oauth2-with-redirect-bearer — a host serving a UI and an API on the same routes, as ArgoCD does with gRPC-web at root paths

Browser chains untouched. Preserving the header costs a UI nothing: browsers never send Authorization, they use the backend's own session cookie.

4. OpenBao's API routed without the errors-redirect plugin

The vault Ingress uses oauth2-with-redirect, whose errors-redirect plugin rewrites any 401-403 into a 302 to the login page. Right for a browser, wrong for an API: OpenBao answers "missing client token" and "permission denied" with 403, and a CLI then gets an HTML page where it expected JSON. A second vault-api Ingress routes /v1 through oauth2-no-redirect — same forwardauth, nothing exempted, but status codes instead of redirects. Traefik derives router priority from rule length when unset, so it carries an explicit priority to beat /.

Security

Nothing is exempted from authentication — every request still passes the same forwardauth, it just gains a second way to prove identity. Nor does this widen who gets in: oauth2-proxy runs email_domains=["*"] with no allowed_groups, so it already authorizes anyone Dex admits, and Dex only admits members of the configured GitHub orgs.

Three things that argument does not cover, worth stating rather than leaving implied:

• It is an argument about who, not about what a stolen credential reaches. toolbox is the first public Dex client on this platform whose audience anything honors — kubectl is public but appears in no audience list anywhere. One secretless, portable token now opens the edge for every host behind it, ArgoCD at the holder's RBAC, and OpenBao at their reader/editor policy. Previously those were three separate credentials, and the edge one was an HttpOnly cookie scoped to the domain.
• Revocation is coarse. Dex id_tokens are stateless. offline_access refresh tokens have no configured expiry (rotation.go:230-233 — zero disables it), so a holder renews indefinitely until the Dex pod restarts, which logs out every user at once. Worth adding expiry.refreshTokens here; not in this PR.
• dex.github.orgs is the only gate, and it is optional. {{- if .Values.dex.github.orgs }} means a cluster rendered with an empty list never reaches the org check at all, and Dex admits any GitHub account — which this change makes considerably more valuable to an attacker. Pre-existing. Fixed in fix: fail the render when dex.github.orgs is empty #1488 — note it needs fail guarded by not, not required, which rejects only nil and the empty string and lets an empty list through.

skipApprovalScreen: true is likewise pre-existing, and means a device-code grant shows the victim no page naming what is being granted. Unchanged here, but its blast radius is now larger.

Using it

./toolbox up <captain-domain>
./toolbox argocd app list
./toolbox bao kv get secret/my-app

GlueOps/toolbox is the client half. Neither CLI works with a bare token: argocd needs it in both ARGOCD_AUTH_TOKEN (which becomes Token:, for ArgoCD) and --header "Authorization: Bearer …" (for the edge), and bao cannot place its -header flag reliably at all, so toolbox proxies it. See #63 and #68.

Testing

Installed on nonprod.jupiter.onglueops.rocks via the captain_utils custom-directory flow. All 29 Argo CD Applications Synced/Healthy; oauth2-proxy, glueops-dex and argocd-server all rolled their pods, so the config is live.

oauth2-proxy now attempts bearer verification — the load-bearing check. Before, the header was ignored outright and nothing was logged from jwt_session.go:

$ curl -H 'Authorization: Bearer not.a.jwt' https://argocd.<domain>/api/version
401
# oauth2-proxy log:
[jwt_session.go:54] Error retrieving session from token in Authorization header: no valid bearer token found

Routing and regressions:

argocd  /                       -> 302   browsers still redirected to login
argocd  /api/v1/session         -> 401   CLI gets a status, not HTML
vault   /                       -> 302   UI intact
grafana /                       -> 302   untouched

Retracted test

An earlier version of this description offered POST https://dex.<domain>/device/code client_id=toolbox -> 200, user_code returned as evidence the new Dex client exists. It proves nothing. Dex never validates client_id at that endpoint — handleDeviceCode stores it verbatim and the first GetClient call is in handleDeviceCallback (deviceflowhandlers.go:332), after the browser leg. That request returns 200 identically on main, where toolbox does not exist, and for client_id=doesnotexist.

The falsifiable check is the connector leg. Note /auth itself is not enough: with a single connector configured Dex auto-redirects to it before validating the client, so /auth returns 302 even for a client that does not exist. The lookup happens one hop later, at /auth/<connector-id>:

curl -so /dev/null -w '%{http_code}\n' \
  "https://dex.<domain>/auth/github?client_id=toolbox&redirect_uri=%2Fdevice%2Fcallback&response_type=code&scope=openid+email+groups&state=x"

404 (Invalid client_id) means the client is not there. Verified on a live cluster: toolbox and kubectl return 302, a made-up client id returns 404 with Invalid client_id.

Not verified

The full argocd/bao login round trip through the edge — both need an interactive GitHub browser approval I could not drive headlessly. Every layer they depend on is verified individually above.

Ordering

GlueOps/docs-argocd#63 must not reach a cluster before this does. Traefik drops a router whose middleware is missing, so argocd.<domain> would 404 entirely — fail-closed, self-healing, but an avoidable outage. The tenant README's documented order installs ArgoCD before the platform chart, so on an upgrade it needs reversing: GlueOps/terraform-module-cloud-multy-prerequisites#720.

[venkatamutyala]

Installed and verified on a live cluster

nonprod.jupiter.onglueops.rocks, via the captain_utils custom-directory flow (helm upgrade --install glueops-platform <dir> -f platform.yaml -n glueops-core). All 29 Argo CD Applications are Synced/Healthy afterwards and no pod is out of Running.

Landed in-cluster. oauth2-proxy and glueops-dex both rolled their pods, so the new config is actually live:

$ kubectl get cm -n glueops-core-oauth2-proxy oauth2-proxy -o jsonpath='{.data.oauth2_proxy\.cfg}' | grep -E 'skip_jwt|extra_aud'
skip_jwt_bearer_tokens=true
oidc_extra_audiences=["argocd","vault"]

$ kubectl get middleware -n glueops-core-oauth2-proxy | grep bearer
oauth2-forwardauth-bearer     ...
oauth2-with-redirect-bearer   ...

$ kubectl get middleware -n glueops-core-oauth2-proxy oauth2-forwardauth-bearer -o jsonpath='{.spec.forwardAuth.authResponseHeaders}'
["x-auth-request-user","x-auth-request-email","x-auth-request-access-token"]      # no "authorization"

oauth2-proxy now attempts bearer verification. This is the load-bearing check — before the change the Authorization header was ignored outright and nothing was ever logged from jwt_session.go:

$ curl -H 'Authorization: Bearer not.a.jwt' https://argocd.<domain>/api/version
401

# oauth2-proxy log:
[jwt_session.go:54] Error retrieving session from token in Authorization header: no valid bearer token found in authorization header

A deliberately malformed token is correctly rejected; a real Dex id_token takes the same path and passes.

Routing behaves as intended with the ArgoCD ingresses pointed at the new chains (oauth2-with-redirect-bearer on /, oauth2-api on /api):

/                                -> 302  oauth2/start     # browsers still get a login redirect
/api/v1/session                  -> 401                   # CLI gets a status, not an HTML redirect
/version.VersionService/Version  -> 302  oauth2/start     # unauthenticated; passes with a valid token

Not yet verified

The full argocd login --sso / bao login -method=oidc round trip. Both need an interactive GitHub browser login, which I could not drive headlessly. Every layer they depend on is verified individually above and in GlueOps/terraform-module-kubernetes-hashicorp-vault-configuration#68.