client-registration uses external KEYCLOAK_URL for server-to-server admin calls → unreachable in-cluster on Kind (client-credentials Secret never created)

[cwiklik]

Summary

The client-registration controller uses the external KEYCLOAK_URL (the workload issuer host) for its server-to-server Keycloak Admin API calls. On Kind that host (keycloak.localtest.me) resolves to 127.0.0.1 inside a pod, so the admin-token request fails, no client is registered, and the kagenti-keycloak-client-credentials-<hash> Secret is never created — leaving the injected agent pod stuck FailedMount/Init:0/1, never Ready.

Code trace (internal/controller/clientregistration_controller.go)

• readAuthbridgeConfigMap → KeycloakURL = cm.Data["KEYCLOAK_URL"] (~L434) — this is the external issuer host.
• reconcileOne builds the admin client with that same value (~L261):

  kc := keycloak.Admin{BaseURL: ab.KeycloakURL, HTTPClient: keycloak.DefaultHTTPClient()}
  // → admin token (password grant) + RegisterOrFetchClientWithToken

• The same KEYCLOAK_URL is also consumed by the webhook injector (internal/webhook/injector/namespace_config.go:73, container_builder.go:319) for the workload's JWT validation, where the external issuer is correct.

So one config key serves two incompatible purposes: server-to-server admin (needs in-cluster reachability) vs workload token validation (needs the external issuer).

Evidence

• Live dev cluster (Kind): operator log:

  Keycloak admin token failed ... Post "https://keycloak.localtest.me/realms/master/protocol/openid-connect/token":
  dial tcp 127.0.0.1:443: connect: connection refused
  

  kubectl get secrets -n tx-e2e | grep -c kagenti-keycloak-client-credentials → 0; injected pod Init:0/1, FailedMount: secret "kagenti-keycloak-client-credentials-..." not found.
• CI (release-validation Kind): the token-exchange wait logged Waiting for credentials (0/2) for the full loop; pod never Ready. (Confirm from the operator pod logs in the failure artifact.)

Why it looks "flaky"

It's platform-dependent, not random: on Kind, keycloak.localtest.me → 127.0.0.1 (unreachable in-cluster) → always fails; on OCP, the external route may be reachable from inside the cluster → happens to work. Matches the e2e-kind.yaml "client registration ... flaky/incomplete on Kind" note.

Proposed fix — decouple the admin endpoint from the issuer URL

The admin/registration calls must use an in-cluster-reachable URL, distinct from the external issuer KEYCLOAK_URL used for workload validation:

1. Add KEYCLOAK_INTERNAL_URL to authbridgeConfig + readAuthbridgeConfigMap.
2. In reconcileOne, choose the admin base URL:

   adminBase := ab.KeycloakAdminURL              // KEYCLOAK_INTERNAL_URL
   if adminBase == "" { adminBase = deriveInClusterURL(kcNamespace) }  // http://keycloak-service.<kc-ns>.svc:8080
   if adminBase == "" { adminBase = ab.KeycloakURL }                   // last resort (OCP where route is in-cluster reachable)
   kc := keycloak.Admin{BaseURL: adminBase, HTTPClient: keycloak.DefaultHTTPClient()}

   The operator already reads admin creds from the Keycloak namespace (resolveKeycloakAdminCredentials), so deriving the in-cluster service URL needs no new lookup.
3. Populate KEYCLOAK_INTERNAL_URL in authbridge-config from the platform install + the token-exchange setup (kagenti-system already carries http://keycloak-service.keycloak.svc:8080; the admin path just doesn't use it). Keep KEYCLOAK_URL = external issuer for the injector.

Secondary: surface a clear condition/event on the AgentRuntime when admin-token/registration fails (e.g. ClientRegistrationFailed: keycloak unreachable at <url>) — this was invisible until pod events + operator logs were read manually.

Impact / linkage

Blocks the two exec-based token-exchange E2E tests in rossoctl/rossoctl#2129 (they stay xfail until the Secret is reliably created). Related: this repo #417 (Keycloak client contract), #433 (per-namespace KEYCLOAK_REALM). Prior client-registration fix: rossoctl/rossoctl#2098 (namespaces:patch RBAC → client_not_found) — a different failure in the same path.

Assisted-By: Claude Code