refactor(k8s): split manifests by lifecycle so ArgoCD stops reapplying secrets - #7
Merged
Merged
Conversation
…g secrets
ArgoCD tracked the whole k8s/ tree, and CI commits an image tag into
k8s/app on every build, which triggers a full sync. That meant every
deploy re-applied k8s/app/secret.yaml, overwriting the real cluster
secret with its CHANGEME_ placeholders.
Split the tree by how often a resource legitimately changes:
k8s/app/ per-deploy workloads — the only auto-synced path
k8s/platform/ Postgres/MinIO/Redis + ingress, manual sync
k8s/bootstrap/ secrets, RBAC, namespace, one-shot jobs — untracked
by ArgoCD, applied by hand once (see its README)
Real secrets are now gitignored and live only in the cluster; only
*.example.yaml templates are tracked. Scope the sync pipeline to
k8s/app/** so platform changes don't auto-fire.
Also fix an ambiguous ArgoCD targetRevision: "prod" is both a branch
and a CI-force-pushed tag pointing at a different commit. Fully
qualify it as refs/heads/prod.
All manifest moves are pure renames — no spec content changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ArgoCD tracked the whole
k8s/tree (path: k8s), and CI commits an image tag intok8s/appon every build, which triggers a full sync. So every deploy re-appliedk8s/app/secret.yaml— overwriting the real cluster secret with itsCHANGEME_placeholders.Same exposure for
k8s/infra/*/secret.yaml, the RBAC, the namespace, and the one-shot init Jobs (whose specs are immutable — re-applying a changed one fails).Fix
Split the tree by how often a resource legitimately changes:
k8s/app/attendee, auto via CIk8s/platform/attendee-platform, manualk8s/bootstrap/kubectlby handReal secrets are now gitignored and live only in the cluster; only
*.example.yamltemplates are tracked..woodpecker/sync.yamlis scoped tok8s/app/**so platform changes don't auto-fire.Also: ambiguous ArgoCD ref
prodis both a branch and a tag, pointing at different commits — CI force-pushes the tag every build:targetRevision: prodwas therefore ambiguous. Now fully qualified asrefs/heads/prod. Whether that tag should exist at all is worth deciding separately — it will keep confusing anything that resolves a bareprod.Verification
git diff -M --stat devshows every relocated manifest at| 0— pure renames, zero spec content changed, including the/consoleingress split from dbc4204.Secret/Job/Role/RoleBinding/ServiceAccount/Namespaceremains underk8s/app(this was the bug).attendee-secret,regcred,attendee-bot-launcher, …) still resolves by name.Before this reaches prod
ArgoCD has been overwriting
attendee-secretwith placeholders. After this it stops managing that secret, so whatever is in the cluster becomes permanent — check it first:Once merged and promoted to
prod, apply both Applications:Follow-up
With Vault on the target cluster,
bootstrap/secrets/should become External Secrets OperatorExternalSecretresources — those hold only Vault references, so they can live ink8s/appand sync freely, removing the manual step this split exists to protect.🤖 Generated with Claude Code