Skip to content

fix(envoy-gateway): make module safe for upgrades, pin v1.9.2 - #40

Merged
surrealwolf merged 1 commit into
mainfrom
fix/envoy-gateway-module-upgrade
Oct 5, 2026
Merged

surrealwolf merged 1 commit into
mainfrom
fix/envoy-gateway-module-upgrade

Conversation

@surrealwolf

Copy link
Copy Markdown
Collaborator

Why

terraform/modules/envoy_gateway could not be used to upgrade Envoy Gateway safely:

  • Data-loss landmine: deploy-envoy-gateway.sh deleted all Gateway API CRDs when crd/gateways.gateway.networking.k8s.io didn't contain the string gateway.envoyproxy.io. None of the live clusters' CRDs do (checked on prd), so any re-run would have deleted every Gateway/HTTPRoute/TCPRoute/UDPRoute on prd (game servers + high-command).
  • null_resource.deploy_envoy_gateway had no triggers, so bumping envoy_gateway_version was a no-op.
  • kubectl apply ... | grep -v ... hid apply failures (no pipefail).
  • gateway_api_version was unused and documented as v1.1.0.

What

  • Download install.yaml, split CRDs from the rest; SSA-apply CRDs first and wait for Established, then the controller (EG 1.9 requires the Gateway API 1.6 CRDs before the controller is upgraded, otherwise TCP/UDP routes are silently skipped).
  • Never delete CRDs.
  • Delete the finished eg-gateway-helm-certgen Job before re-apply (its pod template is immutable). certgen does not pass --overwrite, so existing control-plane cert secrets are kept.
  • set -o pipefail, kubectl rollout status for the controller, warning if the installed Gateway API bundle differs from gateway_api_version.
  • triggers on envoy_gateway_version, gateway_api_version and the script hash (no destroy provisioner, so replacement only re-runs the idempotent install).
  • Defaults/example: Envoy Gateway v1.9.2, Gateway API v1.6.1 (EG 1.7.x→GW API 1.4.1, 1.8.x→1.5.1, 1.9.x→1.6.1).
  • docs/GATEWAY_API_SETUP.md: remove the "install standard-install v1.0.0 first" step, which would downgrade/strip the experimental CRDs.

Part of the UPGRADE_PLAN P0 item "Envoy Gateway + Gateway API CRDs". The live upgrade is done stepwise (1.7.5 → 1.8.5 → 1.9.2) using this script, then tfvars are bumped and reconciled with a targeted -refresh=false apply.

Testing

  • terraform validate passes.
  • bash -n passes; CRD/non-CRD splitter verified against v1.7.5, v1.8.5 and v1.9.2 manifests (all documents accounted for).
  • All CRD versions currently in status.storedVersions (incl. tlsroutes v1alpha3, referencegrants v1beta1) are still served in each bundle.

Made with Cursor

The deploy script deleted every Gateway API CRD (and so every Gateway and
Route) when the gateways CRD lacked the string "gateway.envoyproxy.io",
which is true for all current clusters. The null_resource had no triggers,
so version bumps never re-ran it, and a grep in the pipeline masked
kubectl apply failures.

- never delete CRDs; apply CRDs first (SSA), wait Established, then the
  controller (EG 1.9 needs Gateway API 1.6 CRDs before the controller)
- delete the finished certgen Job before re-apply (immutable template;
  certgen keeps existing cert secrets)
- pipefail; wait for controller rollout; warn on bundle mismatch
- triggers on versions + script hash; gateway_api_version now checked
- defaults/example: EG v1.9.2, Gateway API v1.6.1
- GATEWAY_API_SETUP: drop standard-install v1.0.0 step (would downgrade
  the experimental CRDs)

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Cursor Bugbot completed successfully and did not report findings that need human review. No approval policy required additional review.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@surrealwolf
surrealwolf merged commit 4ebd15b into main Oct 5, 2026
5 checks passed
@surrealwolf
surrealwolf deleted the fix/envoy-gateway-module-upgrade branch October 5, 2026 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant