Skip to content

refactor(k8s): split manifests by lifecycle so ArgoCD stops reapplying secrets - #7

Merged
Rahulkaushik01 merged 1 commit into
devfrom
fix/k8s-argocd-lifecycle-split
Jul 21, 2026
Merged

refactor(k8s): split manifests by lifecycle so ArgoCD stops reapplying secrets#7
Rahulkaushik01 merged 1 commit into
devfrom
fix/k8s-argocd-lifecycle-split

Conversation

@hd1801

@hd1801 hd1801 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Problem

ArgoCD tracked the whole k8s/ tree (path: k8s), and CI commits an image tag into k8s/app on every build, which triggers a full sync. So every deploy re-applied k8s/app/secret.yaml — overwriting the real cluster secret with its CHANGEME_ placeholders.

Same exposure for k8s/infra/*/secret.yaml, the RBAC, the namespace, and the one-shot init Jobs (whose specs are immutable — re-applying a changed one fails).

Fix

Split the tree by how often a resource legitimately changes:

Path Lifecycle Synced by
k8s/app/ every deploy ArgoCD attendee, auto via CI
k8s/platform/ occasionally (Postgres/MinIO/Redis, ingress) ArgoCD attendee-platform, manual
k8s/bootstrap/ once per namespace (secrets, RBAC, namespace, init jobs) nothing — kubectl by hand

Real secrets are now gitignored and live only in the cluster; only *.example.yaml templates are tracked. .woodpecker/sync.yaml is scoped to k8s/app/** so platform changes don't auto-fire.

Also: ambiguous ArgoCD ref

prod is both a branch and a tag, pointing at different commits — CI force-pushes the tag every build:

e692d606 refs/heads/prod
6a0f2815 refs/tags/prod     <- different commit

targetRevision: prod was therefore ambiguous. Now fully qualified as refs/heads/prod. Whether that tag should exist at all is worth deciding separately — it will keep confusing anything that resolves a bare prod.

Verification

  • git diff -M --stat dev shows every relocated manifest at | 0pure renames, zero spec content changed, including the /console ingress split from dbc4204.
  • No Secret/Job/Role/RoleBinding/ServiceAccount/Namespace remains under k8s/app (this was the bug).
  • All YAML parses; all six CI-sed paths still resolve; every cross-directory reference (attendee-secret, regcred, attendee-bot-launcher, …) still resolves by name.
  • Ignore rules verified: real secrets ignored, examples tracked.

Before this reaches prod

ArgoCD has been overwriting attendee-secret with placeholders. After this it stops managing that secret, so whatever is in the cluster becomes permanent — check it first:

kubectl -n meeting-utility get secret attendee-secret -o jsonpath='{.data.DJANGO_SECRET_KEY}' | base64 -d

Once merged and promoted to prod, apply both Applications:

kubectl apply -f argocd/application.yaml -f argocd/application-platform.yaml

Follow-up

With Vault on the target cluster, bootstrap/secrets/ should become External Secrets Operator ExternalSecret resources — those hold only Vault references, so they can live in k8s/app and sync freely, removing the manual step this split exists to protect.

🤖 Generated with Claude Code

…g secrets

ArgoCD tracked the whole k8s/ tree, and CI commits an image tag into
k8s/app on every build, which triggers a full sync. That meant every
deploy re-applied k8s/app/secret.yaml, overwriting the real cluster
secret with its CHANGEME_ placeholders.

Split the tree by how often a resource legitimately changes:

  k8s/app/       per-deploy workloads — the only auto-synced path
  k8s/platform/  Postgres/MinIO/Redis + ingress, manual sync
  k8s/bootstrap/ secrets, RBAC, namespace, one-shot jobs — untracked
                 by ArgoCD, applied by hand once (see its README)

Real secrets are now gitignored and live only in the cluster; only
*.example.yaml templates are tracked. Scope the sync pipeline to
k8s/app/** so platform changes don't auto-fire.

Also fix an ambiguous ArgoCD targetRevision: "prod" is both a branch
and a CI-force-pushed tag pointing at a different commit. Fully
qualify it as refs/heads/prod.

All manifest moves are pure renames — no spec content changed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rahulkaushik01
Rahulkaushik01 merged commit 2902166 into dev Jul 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants