Repository navigation
build(k8s): rehearse the production install and keep client addresses - #64
Merged
Merged
Conversation
The gateway trusts Traefik (TRUSTED_PROXIES 10.42.0.0/16), but on k3d it logged every client as 10.42.0.1: ServiceLB SNATs connections before they reach Traefik under the default externalTrafficPolicy (Cluster), so Traefik never sees the client and all users share one rate-limit bucket (finding #52). deployments/k8s/cluster/traefik-config.yaml is a k3s HelmChartConfig for the Traefik chart k3s ships, setting externalTrafficPolicy: Local. It configures the cluster, not the application, so it sits outside the overlays and is applied once to a new cluster (B). Measured on a second k3d cluster with the production overlay: the Traefik Service reports externalTrafficPolicy=Local and the gateway logs the connection's real source, 172.21.0.4 (k3d's proxy container in front of the node), where the local cluster still logs 10.42.0.1. On the VPS that source is the client itself; the final check is B4. Plan: A10 Refs: Bulgu #52 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two settings belong to the cluster, not to the overlays, and are easy to miss on a fresh VPS: k3s's encryption at rest for Secrets, which is off by default (k3d's default cluster reports "Encryption Status: Disabled"; a cluster created with --secrets-encryption reports Enabled), and the Traefik HelmChartConfig that keeps the client's address. Both were checked on the rehearsal cluster. Plan: A10 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records the production overlay rehearsal on a second k3d cluster (Secret created by hand, the commit's GHCR images, secrets encryption on): 10/10 pods first time and /health 200 with the commit SHA through Traefik. Then the A10 checklist, measured through Traefik: no failed request during gateway and leaderboard rollouts (1050/1050 and 1032/1032), 1001 on a websocket rollout, Redis down handled by readiness without restarts, a config change rolling every Deployment and undone, a drain not blocked by the PDB, a logged-out token refused on /ws, an idle WebSocket kept open by the server's pings, and the client address kept with externalTrafficPolicy Local (final check B4). New: #74 Secrets not encrypted at rest in k3s by default, #75 stale generated Secrets in the local cluster. Notes for D (rollback by redeploying, not rollout undo) and a post-phase item (503 for a missing upstream). Plan: A10 Refs: Bulgu #29, Bulgu #30, Bulgu #31, Bulgu #35, Bulgu #52, Bulgu #74, Bulgu #75 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A9: records #63's CI (a gitleaks false positive on a test fixture, cleared without rewriting history, then 24 checks green) and the measurements on how Secrets are stored (base64 only, no encryption at rest in k3d's default k3s, unreadable to the services' account). #58: after Docker Desktop was reopened, all 14 compose containers came back on their own with restart: always, the behaviour A2a expected and unless-stopped did not give. Plan: A9 Refs: Bulgu #58, Bulgu #74 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A10: the production overlay rehearsed on a second k3d cluster (k3s v1.35.5 with
--secrets-encryption), and the A10 checklist measured through Traefik.deployments/k8s/cluster/traefik-config.yaml: a k3sHelmChartConfigsetting Traefik'sexternalTrafficPolicy: Local. With the default (Cluster) ServiceLB SNATs connections and the gateway saw every client as10.42.0.1(one shared rate-limit bucket, Bulgu chore: Bump grafana/grafana from 11.2.0 to 11.6.16 #52); withLocalit logs the connection's real source. Cluster-level, applied once per new cluster.--secrets-encryption, off by default; the Traefik config)./health200 with the commit SHA through Traefik) and the checklist: gateway rollout 1050/1050 and leaderboard rollout 1032/1032 requests OK, WS close 1001 on a websocket rollout, Redis down handled by readiness without restarts, a ConfigMap change rolling all 7 Deployments and undone, a drain not blocked by the PDB (37 s), a logged-out token refused on/ws, an idle WebSocket kept open by server pings. Real client IP is confirmed on the VPS (B4). New findings build(k8s): add the Let's Encrypt issuers (C3) #74 (no encryption at rest by default), docs(plan): C3 and C4 done; the API serves HTTPS #75 (stale generated Secrets locally).Plan: A10, A9 · Refs: Bulgu #52, #58, #74, #75
🤖 Generated with Claude Code