security: verify ALB OIDC JWT signatures; org-scope pg_stat_activity - #1155
Merged
Conversation
Admin SSO: the admin API trusted the ALB-injected X-Amzn-Oidc-Data header without verifying its signature, and accepted the unsigned X-Amzn-Oidc-Identity header as a fallback. Anything that reaches the control-plane pod without crossing the ALB (same-namespace pod, kubectl port-forward) could forge an operator identity, including admin. The admin layer now verifies the ES256 signature against the ALB's regional AWS public-key endpoint and checks exp, signer, iss, and client claims. The unsigned identity header is no longer consulted. SSO is configured with DUCKGRES_ADMIN_SSO_ISSUER / _CLIENT_ID / _REGION; without the issuer env, SSO headers are ignored and only bearer tokens authenticate. pg_stat_activity: on the multitenant control plane, a full-power org principal (nil QueryAccessPolicy) received every connection on the shared process, including other orgs' query text, usernames, and client addresses. Rows are now always scoped to the caller's org; project-scoped users keep the stricter per-username filter. Standalone mode is unchanged (empty orgIDs match). The mw-dev e2e forged-SSO test now asserts 401 for unsigned headers. Co-authored-by: Shelley <shelley@exe.dev>
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: neutral or increased No coverage-reduction warnings detected. |
jghoman
approved these changes
Sep 3, 2026
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes from a security review of the managed-warehouse query layer:
X-Amzn-Oidc-Dataheader without verifying its signature, and accepted the unsignedX-Amzn-Oidc-Identityheader as a fallback. The control-plane pod is reachable without crossing the ALB (any pod in the namespace perk8s/networkpolicy.yaml, andkubectl port-forwardbypasses NetworkPolicy entirely), so a forged header yielded viewer — or, with a known operator email, admin: org CRUD, tenant root password resets, service-credential mints, worker image pinning, user impersonation.pg_stat_activitycross-tenant disclosure (High). On the multitenant control plane all orgs' connections register in one process-wide map; a full-power org principal (nilQueryAccessPolicy) received every row — other orgs' query text (with literal values), usernames, database names, and client IPs.Changes
controlplane/admin/alb_oidc.go:ALBOIDCVerifierchecks the ES256 signature against the ALB's regional AWS public keys (public-keys.auth.elb.<region>.amazonaws.com/<kid>, cached per kid), plusexp,signer(must be an ALB ARN in the configured region),iss, andclientclaims. Algorithm pinned to ES256; kid charset-restricted before it enters a URL.AuthMiddlewarenow takes the verifier. The unsignedX-Amzn-Oidc-Identityheader is no longer consulted. WithoutDUCKGRES_ADMIN_SSO_ISSUERthe SSO path is off entirely (fail closed): only bearer tokens authenticate; a startup warning is logged. Region falls back toDUCKGRES_AWS_REGION.visiblePgStatActivityConnsnow always filters byorgID; project-scoped users keep the stricter per-username filter. Standalone mode is unchanged (empty orgIDs match).tests/mw-dev/e2e/harness.sh): the forged-SSO test now asserts 401 for unsignedX-Amzn-Oidc-Dataand forX-Amzn-Oidc-Identity.Deployment ordering
The companion charts PR sets
DUCKGRES_ADMIN_SSO_ISSUER/DUCKGRES_ADMIN_SSO_CLIENT_ID(derived from the existingadminUI.ingress.cognitovalues) on the control-plane Deployment. The charts change must be synced in each environment before this image rolls there — otherwise SSO login 401s (break-glass token login still works). Dev auto-deploys on merge, so merge the charts PR first and confirm Argo sync.Testing
pg_stat_activity: org root sees exactly its own org's connections; standalone sees all.go test -tags kubernetes ./...green for all affected packages (tests/integrationandtests/trino-ducklake-smokefail identically on main in this sandbox — they need Docker services).Rollback
Revert the image pin via the state pipeline; with the charts envs in place the old image simply ignores them.