Skip to content

[sec-check] fix: remove hardcoded kubestellar console secrets and auto-applied manifest - #90

Closed
Danathar wants to merge 7 commits into
projectbluefin:mainfrom
Danathar:sec/fix-kubestellar-console-secrets
Closed

Danathar wants to merge 7 commits into
projectbluefin:mainfrom
Danathar:sec/fix-kubestellar-console-secrets

Conversation

@Danathar

@Danathar Danathar commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Security Fix

Addresses finding in #72: files/k0s/manifests/kubestellar/01-kubestellar-console-github-oauth.yaml committed dummy GitHub OAuth credentials and a static JWT secret that was automatically seeded into /var/lib/k0s/manifests/ and deployed by the k0s manifest deployer on default installations.

Changes

  1. Remove Auto-Applied OAuth Secret Manifest: Deleted files/k0s/manifests/kubestellar/01-kubestellar-console-github-oauth.yaml so no hardcoded secrets or dummy credentials are baked into the sysext or auto-applied by k0s.
  2. Make JWT_SECRET Optional in Console Deployment: In files/k0s/manifests/kubestellar/40-kubestellar-console.yaml, set optional: true on the JWT_SECRET Secret reference (matching the existing optional: true on client-id and client-secret). When unset, the console server generates a random secret at startup because the manifest sets DEV_MODE=true (pkg/api/server.go → loadOrCreateDevSecret, persisted under os.UserConfigDir() inside the container). Without DEV_MODE an unset JWT_SECRET is a fatal exit rather than a weak default, so dropping DEV_MODE requires populating jwt-secret in the Secret.
  3. Regression Tests: Updated tests/unit/test_k0s_manifests.py to assert 01-kubestellar-console-github-oauth.yaml is not present in manifests, and updated tests/unit/test_kubestellar_kiosk.py to verify all Secret references are optional.
  4. Documentation: Documented how operators can configure production GitHub OAuth credentials and a persistent JWT secret via an operator-created Secret in docs/skills/k0s-sysext-ops.md.

Closes #72

— hive: backend=agy

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good fix. Two things to confirm before it lands, and one much larger issue in the same file.

Deleting 01-kubestellar-console-github-oauth.yaml is clearly right — jwt-secret: default-jwt-secret-123456789012345678901234 was a fixed signing key auto-applied on every default install, and shipping it in the sysext meant every Bluefin Server signed tokens with the same value. Removing it is the correct call.

I checked the obvious way this could break and it does not: deleting that file does not orphan the namespace, because 40-kubestellar-console.yaml declares it itself.

$ git show pr90:files/k0s/manifests/kubestellar/40-kubestellar-console.yaml | head -4
apiVersion: v1
kind: Namespace
metadata:
  name: kubestellar-console

So the Service and Deployment still have a namespace to land in. Good.

1. The central safety claim is not verifiable from this repo

The body states:

When unset, the console container automatically generates an ephemeral random 32-byte secret at startup (console/start.sh).

console/start.sh is not in this repository:

$ git ls-tree -r --name-only pr90 | grep -i 'start.sh'
(no output)

It lives inside ghcr.io/kubestellar/console:v0.3.34. The whole security argument for optional: true rests on that behaviour. If the image instead falls back to an empty or compiled-in default signing key when JWT_SECRET is unset, this change makes forgery easier than the status quo, because the bad-but-known secret is replaced by a bad-and-silent one. Please link the upstream start.sh line, or pin the behaviour with a comment in the manifest, so a future image bump can't quietly invalidate it.

2. The console image is on a mutable tag

image: ghcr.io/kubestellar/console:v0.3.34

No digest. This is the same class of issue #81 is fixing for argocd, and the kiosk proxy in the sibling file is already pinned (nginx@sha256:62223d644fa234c3a1cc785ee14242ec47a77364226f1c811d2f669f96dc2ac8). Since this PR is already editing this file for supply-chain reasons and the behaviour in item 1 is image-dependent, pinning by digest here would be cheap and would make the claim durable.

3. The bigger hole is three lines above the one you changed

Unchanged by this PR, but in the same container spec:

        env:
        - name: DEV_MODE
          value: "true"
        - name: ALLOW_DEV_MODE_IN_CLUSTER
          value: "true"
        - name: DEV_USER_LOGIN
          value: "admin"

To be explicit about baseline: this is pre-existing on main, not introduced here. But it substantially undercuts what this PR achieves. Hardening the JWT signing key matters only if authentication is enforced; with dev mode explicitly enabled in-cluster and a hardcoded admin login, anyone who reaches the console is already an administrator, regardless of how the JWT is signed. Worth a follow-up issue — and it's arguably the finding that should be at the top of #72's queue.

CI status

No checks have run on this PR at all:

mcp__github__pull_request_read(method="get_check_runs", pullNumber=90)
  -> {"total_count":0,"check_runs":[]}

mergeable_state is unstable; it's a fork PR (Danathar/server) and looks like it needs maintainer approval for workflows to run. I ran the gates locally on main + this branch: 201 passed, 1 xfailed, 41/41 bats, all three .github/scripts checkers exit 0 — identical to the main baseline at 7f376fa. Nothing regressed. The two new assertions are cheap and correctly aimed.

Merge-order note

This and #91 both edit docs/skills/k0s-sysext-ops.md and tests/unit/test_kubestellar_kiosk.py. Each merges cleanly into main alone, but not into each other:

$ git merge pr90   # onto main+pr91
CONFLICT (content): Merge conflict in docs/skills/k0s-sysext-ops.md

Whichever goes second needs a rebase.

Marking this as a comment rather than a block — items 1 and 2 are worth resolving first, but the change itself is an improvement over what's on main.


Generated by Claude Code

Danathar added a commit to Danathar/server that referenced this pull request Sep 12, 2026
hanthor's review on projectbluefin#90 confirmed the optional: true change is correct
but flagged two things to resolve first:

1. The safety argument ("the console generates an ephemeral random
   secret when JWT_SECRET is unset") rests on upstream behaviour this
   repo can't see. Fetched kubestellar/console's start.sh at the exact
   pinned tag (v0.3.34) and confirmed it directly: when JWT_SECRET is
   unset, it generates `openssl rand -hex 32` (falling back to
   /dev/urandom, never an empty or compiled-in default), persists it to
   $INSTALL_DIR/data/.jwt_secret so it survives a plain container
   restart, and exports it. Since this Deployment mounts no volume for
   that path, it's lost on pod recreation -- so "ephemeral" is accurate
   at the pod-lifetime granularity, just not on every single restart.
   Added a comment citing the exact upstream lines so a future image
   bump that changes this can't invalidate the claim silently.

2. The image was on a mutable tag while the sibling kiosk-proxy nginx
   image in the same manifest set is already digest-pinned. Resolved
   ghcr.io/kubestellar/console:v0.3.34's current digest via skopeo
   (confirmed it's an OCI image index, same multi-arch-safe shape as a
   tag reference) and pinned it.

Left item 3 (DEV_MODE/admin login already on main) alone, per the
review's own framing -- pre-existing, out of scope, worth a follow-up
issue rather than scope creep here.

Verified: all 8 relevant unit tests pass, full suite 201/201 + 1
xfailed, YAML parses clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
@Danathar

Copy link
Copy Markdown
Contributor Author

@hanthor — pushed in 2de9d42:

  1. Verified the upstream claim directly. Fetched kubestellar/console's start.sh at the exact pinned tag (v0.3.34): when JWT_SECRET is unset, it runs openssl rand -hex 32 (falling back to /dev/urandom, never an empty/compiled-in default), persists it to $INSTALL_DIR/data/.jwt_secret, and exports it. Since this Deployment mounts no volume for that path, it survives a plain container restart but is lost on pod recreation — so "ephemeral" holds at pod-lifetime granularity. Added a comment citing the exact upstream lines (https://github.com/kubestellar/console/blob/v0.3.34/start.sh#L524-L541) so a future image bump that changes this doesn't invalidate the claim silently.

  2. Pinned the image by digest — resolved ghcr.io/kubestellar/console:v0.3.34's current digest via skopeo (it's an OCI image index, same multi-arch-safe shape a tag gives you) and pinned it, matching the kiosk-proxy nginx image's existing pattern in the sibling manifest.

Left item 3 alone per your own framing — pre-existing on main, worth a follow-up issue rather than scope creep here.

Verified: all 8 relevant unit tests pass, full suite 201/201 + 1 xfailed.

On the merge-order note with #91: pushed both independently; whichever lands second will need the small rebase on docs/skills/k0s-sysext-ops.md / tests/unit/test_kubestellar_kiosk.py you flagged.

@castrojo

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

The branch could not be refreshed onto current main automatically because tests/unit/test_k0s_manifests.py conflicts. Please rebase and resolve that file. Current main includes #149, so the refreshed branch will use the shared Project Bluefin QEMU installer gate.

Danathar and others added 2 commits September 14, 2026 19:04
…ed manifest

Delete files/k0s/manifests/kubestellar/01-kubestellar-console-github-oauth.yaml
so no hardcoded JWT secrets or dummy OAuth credentials are automatically seeded
into /var/lib/k0s/manifests/ or applied by the k0s manifest deployer.

Make JWT_SECRET secretKeyRef in 40-kubestellar-console.yaml optional so the
console starts cleanly in local development mode without requiring a pre-seeded
Secret manifest (in-container console startup generates an ephemeral secret
when unset).

Update unit test contracts in test_k0s_manifests.py and test_kubestellar_kiosk.py
to assert the secret manifest is not present and all secret references are optional.

Document operator-provided OAuth Secret configuration in docs/skills/k0s-sysext-ops.md.

Closes projectbluefin#72

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
hanthor's review on projectbluefin#90 confirmed the optional: true change is correct
but flagged two things to resolve first:

1. The safety argument ("the console generates an ephemeral random
   secret when JWT_SECRET is unset") rests on upstream behaviour this
   repo can't see. Fetched kubestellar/console's start.sh at the exact
   pinned tag (v0.3.34) and confirmed it directly: when JWT_SECRET is
   unset, it generates `openssl rand -hex 32` (falling back to
   /dev/urandom, never an empty or compiled-in default), persists it to
   $INSTALL_DIR/data/.jwt_secret so it survives a plain container
   restart, and exports it. Since this Deployment mounts no volume for
   that path, it's lost on pod recreation -- so "ephemeral" is accurate
   at the pod-lifetime granularity, just not on every single restart.
   Added a comment citing the exact upstream lines so a future image
   bump that changes this can't invalidate the claim silently.

2. The image was on a mutable tag while the sibling kiosk-proxy nginx
   image in the same manifest set is already digest-pinned. Resolved
   ghcr.io/kubestellar/console:v0.3.34's current digest via skopeo
   (confirmed it's an OCI image index, same multi-arch-safe shape as a
   tag reference) and pinned it.

Left item 3 (DEV_MODE/admin login already on main) alone, per the
review's own framing -- pre-existing, out of scope, worth a follow-up
issue rather than scope creep here.

Verified: all 8 relevant unit tests pass, full suite 201/201 + 1
xfailed, YAML parses clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
@Danathar
Danathar force-pushed the sec/fix-kubestellar-console-secrets branch from 2de9d42 to 63ded6d Compare September 14, 2026 23:06
@castrojo

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Update: the second refresh onto main completed successfully after the shared KVM fix landed. No author action is needed for the branch refresh.

castrojo
castrojo previously approved these changes Sep 18, 2026

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The secret is genuinely removed rather than relocated, and both of @hanthor's asks are closed.

First, the thing that was actually blocking this PR was not review — it was that CI had never run. This is a fork PR from a contributor with read access, and build.yml triggers on plain pull_request while checking out the PR head, so its runs sat at conclusion: action_required with zero jobs dispatched. gh pr checks showed literally nothing, which reads identically to "not yet pushed". I have approved the workflow runs; unit and docs now pass and build is in progress. Worth knowing for future fork PRs from the same contributor.

Verified: it removes the secret, it does not move it. A repo-wide search for default-jwt-secret returns exactly one hit, and it is files/k0s/manifests/kubestellar/01-kubestellar-console-github-oauth.yaml — the file this PR deletes. No copy of the value appears anywhere else, and the diff introduces none. The replacement documentation in docs/skills/k0s-sysext-ops.md uses only placeholders (<github-client-id>, <github-client-secret>, <random-32-byte-or-longer-secret>) and tells operators to create the Secret themselves at runtime, so nothing is carried over into docs. A search for references to the deleted filename returns zero, so nothing else in the tree points at the removed manifest, and deleting it does not orphan the namespace — 40-kubestellar-console.yaml declares kind: Namespace / name: kubestellar-console itself.

That matters because the file was auto-applied by the k0s manifest deployer on every default install, so this was a fixed, published signing key on every machine, not a sample.

Both review asks are addressed. The optional: true safety argument now carries an inline citation to the upstream source it rests on (console/start.sh at v0.3.34, showing openssl rand -hex 32 persisted to $INSTALL_DIR/data/.jwt_secret, never an empty or compiled-in default), and the image is pinned by digest rather than a mutable tag. The digest pin matches the existing nginx@sha256:... pattern in the sibling kiosk-proxy manifest, so it is consistent with how this repo already does it.

Two things for the record, neither blocking.

assert "optional: false" not in console in tests/unit/test_kubestellar_kiosk.py is a whole-file negative assertion. It passes today, but it will fire spuriously the first time any future secret reference in that manifest legitimately needs to be mandatory. Scoping it to the JWT_SECRET block would make it durable. Not worth holding the PR for.

More importantly, and explicitly not this PR's fault: three lines above the changed line, untouched here and still on main, sit DEV_MODE: "true", ALLOW_DEV_MODE_IN_CLUSTER: "true" and DEV_USER_LOGIN: "admin". Hardening the JWT signing key is worth doing, but its value is capped while anyone who reaches the console arrives as admin. You were right to decline to scope-creep into it here. I think that deserves its own issue and is arguably more urgent than the one this PR closes — I would rather that be tracked than quietly inherited.

Approving. Needs one more independent approval under the two-reviewer policy. Merge should wait for build to report green and for the /var regression on main (#173) so this lands on a base where installer-test means something.

The whole-file negative `assert "optional: false" not in console` forbids
the string anywhere in 40-kubestellar-console.yaml, so any future mandatory
secret or configmap reference would trip it spuriously. Parse the manifest
with yaml.safe_load_all, select the Deployment, and assert the JWT_SECRET
env entry's secretKeyRef.optional is True — narrower and stronger, since it
checks the actual field on the actual reference.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
castrojo previously approved these changes Sep 18, 2026

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving. My earlier approval was dismissed when I pushed a fix to this branch; this restores it and explains the change, since I am not the author.

What I changed (commit bec017a, 6d47b37..bec017a, one file, tests/unit/test_kubestellar_kiosk.py).

The new test carried a whole-file negative assertion:

assert "optional: false" not in console

where console is the entire text of 40-kubestellar-console.yaml — a three-document manifest (Namespace, Service, Deployment). It passed, but it forbade the substring optional: false anywhere in all three documents, which is much broader than the intent. The first time any future secret or configMap reference in that file legitimately needed to be mandatory, it would fire and read as a regression.

Replaced with a parsed assertion on the actual field:

deployment = next(doc for doc in yaml.safe_load_all(...) if doc and doc.get("kind") == "Deployment")
jwt_secret = next(env for env in deployment["spec"]["template"]["spec"]["containers"][0]["env"] if env["name"] == "JWT_SECRET")
assert jwt_secret["valueFrom"]["secretKeyRef"]["optional"] is True

That is both narrower and stronger — it checks the real field on the real reference rather than the absence of a string. Parsing matches the sibling tests/unit/test_k0s_manifests.py, which already uses yaml.safe_load_all on manifest text, so it follows existing convention rather than introducing a new one.

Proof the new assertion is not vacuous, both directions:

  • Flipping the JWT_SECRET reference to optional: false fails it: assert False is True, tests/unit/test_kubestellar_kiosk.py:110: AssertionError. Manifest restored afterwards.
  • Adding an unrelated mandatory reference elsewhere in the same Deployment (CONSOLE_BRANDING with configMapKeyRef ... optional: false) trips the OLD assertion spuriously but passes the new one. That is the exact failure this change prevents.

Verification. python3 -m pytest tests/unit/test_kubestellar_kiosk.py -q gives 5 passed; with test_k0s_manifests.py too, 12 passed. gh pr diff 90 --name-only confirms the manifest is not in the diff — only the test file changed.

The PR itself is unchanged from my earlier review and still right. The secret is genuinely removed rather than relocated: a repo-wide search for default-jwt-secret returns exactly one hit, the file this PR deletes, and the replacement docs use only placeholders. Both of @hanthor's asks are closed — the optional: true argument now cites upstream console/start.sh at v0.3.34, and the image is digest-pinned.

Still not this PR's problem but worth its own issue, as I noted before: DEV_MODE: "true", ALLOW_DEV_MODE_IN_CLUSTER: "true" and DEV_USER_LOGIN: "admin" remain three lines above, untouched on main. Hardening the JWT signing key is capped in value while anyone reaching the console arrives as admin.

CI: unit and docs pass; build is running. The only expected red is installer-test, which fails on main itself and is unrelated.

Needs one more independent approval under the two-reviewer policy.

@hivecommons-hive hivecommons-hive Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • low — the manifest comment and PR body cite the wrong fallback mechanism. files/k0s/manifests/kubestellar/40-kubestellar-console.yaml:65-72 (and the PR body) say the image's start.sh generates a random JWT secret when JWT_SECRET is unset, persisted to $INSTALL_DIR/data/.jwt_secret. But the container's entrypoint is entrypoint.sh (Dockerfile at v0.3.34, ENTRYPOINT ["./entrypoint.sh"]), which never runs start.sh — that script is the local quick-start installer. The actual fallback is in the Go server: pkg/api/server.go:81-89 generates a random secret only when DEV_MODE=true (via loadOrCreateDevSecret, pkg/api/startup.go:359-387, persisted under os.UserConfigDir(), not data/.jwt_secret); in non-dev mode an unset JWT_SECRET is a fatal exit. The runtime outcome here is as intended because this manifest sets DEV_MODE=true (40-kubestellar-console.yaml:44-45), and the failure mode without it is a safe crash rather than a weak secret — so this is non-blocking. But the comment should be corrected: an operator removing DEV_MODE=true while trusting the comment's "start.sh generates a secret" claim will get a crash-looping console, and the cited persistence path is wrong.

Verified clean: the deleted 01-...-github-oauth.yaml Namespace is redundantly created by 40-kubestellar-console.yaml:1-4, so removal orphans nothing; the pinned digest sha256:f27ada… resolves to the v0.3.34 tag on ghcr.io; pyyaml is already installed by .github/workflows/unit-tests.yml, so the new import yaml in tests/unit/test_kubestellar_kiosk.py:5 is fine.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar

Copy link
Copy Markdown
Contributor Author

On installer-test: the failure here is not this PR. The QEMU boot logs Timed out waiting for device /dev/disk/by-partlabel/var, every unit that needs /var fails as a dependency, and the console readiness wait then times out after 900s. Every Build DDI artifacts run in the repo since 2026-09-15 fails the same way, including main. mrbobbytables has #173 open for the /var mount deadlock; once that lands this PR's build should be re-run rather than read as a regression here.

…comment

The comment said the image's start.sh generates a random secret when
JWT_SECRET is unset and persists it under $INSTALL_DIR/data/.jwt_secret.
At the pinned v0.3.34 the image entrypoint is entrypoint.sh, which never
runs start.sh (that script is the local quick-start installer). The fallback
that actually applies is in the Go server: with DEV_MODE=true it calls
loadOrCreateDevSecret and persists .jwt-secret under os.UserConfigDir();
without DEV_MODE an unset JWT_SECRET is a fatal exit. Say so, so an operator
who drops DEV_MODE knows to set jwt-secret in the Secret instead of expecting
a generated one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VXwvRu29c2MkwftzkgAHcC
@Danathar

Copy link
Copy Markdown
Contributor Author

Pushed a0c8829 to correct the JWT_SECRET fallback description, and updated item 2 of the PR body to match. The Hive finding checks out against v0.3.34: the image ENTRYPOINT is entrypoint.sh, and the fallback that applies is loadOrCreateDevSecret in pkg/api/server.go, which only runs under DEV_MODE=true; otherwise an unset JWT_SECRET is a fatal exit. Comment-only change, unit suite still 207 passed.

@castrojo sorry for the churn: this push will have dismissed your approval again. The runtime behaviour is unchanged from what you reviewed.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • (info) The diff also switches the console image from tag v0.3.34 to a digest pin (files/k0s/manifests/kubestellar/40-kubestellar-console.yaml:43), which the PR body's list of changes doesn't mention. I verified the digest resolves to v0.3.34 on ghcr.io (sha256:f27adaf…be32), so it's benign — just note it in the description.

Verified while reviewing: the deleted 01-…-github-oauth.yaml also created the kubestellar-console Namespace, but 40-kubestellar-console.yaml:1-4 creates it too, so nothing breaks; and upstream kubestellar/console@v0.3.34 pkg/api/server.go:81-89 confirms DEV_MODE generates a random JWT secret while non-DEV_MODE fatals on unset JWT_SECRET, matching the PR body.

No findings from: correctness, security, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor

Closing as superseded by #210.

`files/k0s/manifests/kubestellar/` is deleted, and #210 fixes #72 more thoroughly than a manifest edit could:

  • `files/cluster/manifests/console/auth-secret.yaml` mints a random JWT secret inside the cluster on first boot, once, idempotently. Nothing is committed, and no two installs share a secret.
  • `files/cluster/manifests/kubeflex/postgres-secret.yaml` does the same for the PostgreSQL superuser password (32 random chars from `/dev/urandom`), so it never appears in a pod spec either.
  • The OAuth client secret is read from a Secret key rather than a committed dummy.

Thanks for filing this — the in-cluster generator pattern came straight out of your analysis.

@castrojo castrojo closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[sec-check] kubestellar console ships hardcoded JWT secret + dummy GitHub OAuth credentials, auto-applied by default

3 participants