fix(infra): skip envoy-gateway's bundled CRDs, they conflict with our own - #23
Merged
Merged
Conversation
… own envoy-gateway-stage-do-atl1 has failed to sync since yesterday, retrying 270+ times: customresourcedefinitions.apiextensions.k8s.io "grpcroutes..." is forbidden: ValidatingAdmissionPolicy 'safe-upgrades...' denied request: Installing experimental CRDs on top of standard channel CRDs is prohibited by default. gateway-helm's bundled CRDs are hardcoded to the "experimental" Gateway API channel (verified against the actual chart - every CRD in its bundle carries that annotation, no values toggle exists to change it, no condition on the crds subchart dependency to disable it either). Stage already has "standard" channel Gateway API CRDs from our own dedicated gateway-api-crds appset (pulls directly from kubernetes-sigs/gateway-api upstream, cloud-agnostic, on every cluster) - envoy-gateway's own copy just conflicts with it. Platform never hit this because its envoy-gateway CRDs happened to install successfully first, so every sync since has been a no-op update rather than a fresh install attempt. Not disabling CRD installation and hoping the CNI already provides them - that would only work by accident on DOKS and could leave EKS with no Gateway API CRDs at all. skipCrds relies on our own portable gateway-api-crds appset instead, which runs on every cluster regardless of cloud/CNI.
Helm template diffdiff -u --recursive --label base --label head base head
--- base
+++ head
@@ -218,7 +218,7 @@
app.kubernetes.io/managed-by: Helm
type: Opaque
data:
- SECRET_KEY_BASE: V3BHYVFHR3k3S0Q3STJzMmFheU9DOWNOQUVxT0puaWZmSmNLd2ZmSDRQN3d2eUx2QnBGRExtSEZlQ3ozR3dWUUZJWThJdkJNQ1o5MW1BWnBjUWxnYjNPUGVk
+ SECRET_KEY_BASE: dGpqV2E4aEF5N2dmRXNmWkd3em1kd20zVmwxTG8wZlc4VVlDdUNHdUZOcGpINGdyZkU2SjdsRDVlMjF2eGNEbGVHZnZsRFJtbTFSNEVscjk2VlhRbHFacWJ0
TOTP_VAULT_KEY: ZHN4dmJuM2p4RGQxNmF6MlFwc1g1QjhPK2xseGpRMlNKRTJpNUJ6eDM4ST0=
DATABASE_URL: cG9zdGdyZXM6Ly9wb3N0Z3Jlczpwb3N0Z3Jlc0BwbGF1c2libGUtYW5hbHl0aWNzLXBvc3RncmVzcWw6NTQzMi9wbGF1c2libGVfZGI=
CLICKHOUSE_DATABASE_URL: aHR0cDovL2NsaWNraG91c2U6cGFzc3dvcmRAcGxhdXNpYmxlLWFuYWx5dGljcy1jbGlja2hvdXNlOjgxMjMvcGxhdXNpYmxlX2V2ZW50c19kYg== |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the infrastructure ApplicationSet to prevent envoy-gateway from installing its bundled Gateway API CRDs, avoiding conflicts with the repo’s dedicated gateway-api-crds installation (standard channel CRDs).
Changes:
- Adds a per-app
skipCrdsflag to the infrastructure ApplicationSet list generator. - Enables
helm.skipCrds: truespecifically forenvoy-gateway, leaving other infrastructure apps unchanged.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…lated
ArgoCD ApplicationSets can't template boolean fields (documented -
"This field may NOT be templated, because it is a boolean field").
skipCrds: {{ .skipCrds }} in the shared infrastructure appset's list
generator broke argocd-validate's raw-YAML parse (unquoted {{ }} reads
as a YAML flow mapping) - and quoting it would have "fixed" that only
to fail for real at Application admission instead
("skipCrds in body must be of type boolean: 'string'", a documented
failure others have hit the same way).
envoy-gateway now gets its own ApplicationSet (same clusters: {}
generator, same value-file/override pattern as the shared one) so
skipCrds: true can be a real, non-templated boolean.
Comment on lines
+17
to
+23
| - repoURL: https://github.com/jonathandieu/dieubernetes | ||
| targetRevision: HEAD | ||
| ref: values | ||
| # chart source: umbrella chart with base values + optional cluster override | ||
| - repoURL: https://github.com/jonathandieu/dieubernetes | ||
| targetRevision: HEAD | ||
| path: charts/infrastructure/envoy-gateway |
Comment on lines
+11
to
+12
| metadata: | ||
| name: "envoy-gateway-{{ .name }}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
envoy-gateway-stage-do-atl1 has failed to sync since yesterday, retrying 270+ times:
Root cause
gateway-helm's bundled CRDs are hardcoded to the "experimental" Gateway API channel (verified against the actual chart tarball - every CRD carries that annotation, no values toggle exists to change it, noconditionon thecrdssubchart dependency to disable it either).Stage already has "standard" channel Gateway API CRDs, from our own dedicated
gateway-api-crdsappset (pulls directly fromkubernetes-sigs/gateway-apiupstream at v1.2.1,config/crd/standard, cloud-agnostic, on every cluster).envoy-gateway's own bundled copy just conflicts with it.Platform never hit this because its
envoy-gatewayCRDs happened to install successfully first (before this repo'sgateway-api-crdsappset existed there), so every sync since has been a no-op update rather than a fresh install attempt.Why this isn't DOKS/Cilium-specific, and why we're keeping envoy-gateway rather than switching to Cilium's own Gateway API
This is a known, acknowledged upstream tension, not a DOKS quirk: envoyproxy/gateway#7238 - Envoy Gateway still defaults to the experimental channel for historical reasons (2022, when standard channel lacked features), and by the maintainers' own words it now "installs unsupported experimental CRDs like ListenerSet into clusters unnecessarily."
DOKS's Cilium ships a working Gateway API implementation for free, but EKS and GKE don't have an equivalent free option (GKE has its own separate, non-Cilium native controller; EKS has no default Cilium at all; AKS's managed Cilium doesn't expose Gateway API config). Since EKS/DOKS/GKE are all real near-term targets for dieu.dev, keeping one portable, self-installed implementation (Envoy Gateway) that works identically on all three is worth the one-time setup cost here, versus juggling a different native Gateway API story per cloud.
Fix
helm.skipCrds: trueon envoy-gateway's Application source, in its own dedicatedenvoy-gatewayApplicationSet rather than a list entry in the sharedinfrastructureone.envoy-gateway needed to move out of the shared appset because ArgoCD ApplicationSets can't template boolean fields (documented: "This field may NOT be templated, because it is a boolean field"). A per-item
skipCrds: {{ .skipCrds }}brokeargocd-validate's raw-YAML parse (unquoted{{ }}reads as a YAML flow mapping), and quoting it would only have traded that for a real Application-admission failure instead ("skipCrds in body must be of type boolean: 'string'"- a documented failure others have hit the same way). Its own appset (sameclusters: {}generator and override-file pattern as the shared one) letsskipCrds: truebe a real, non-templated boolean.Deliberately not disabling CRD installation and hoping the CNI already provides them - that would only work by accident on DOKS and leaves EKS with no Gateway API CRDs if its CNI doesn't provide them.
skipCrdsinstead relies on our own portablegateway-api-crdsappset, which runs on every cluster regardless of cloud/CNI.What we lose vs. what we don't
Only genuinely experimental-only types are affected:
TCPRoute/UDPRoute/TLSRoute/ListenerSet.GRPCRouteis not one of them - it graduated to the standard channel in Gateway API v1.1 (2024) and is already included in ourgateway-api-crdsappset's v1.2.1 bundle (verified directly against the upstream repo'sconfig/crd/standarddirectory at that tag). Envoy Gateway's ownGatewayobject already listsGRPCRouteas a supported kind on both listeners today.