Skip to content

fix: keep the NetBird daemon socket and log file off the read-only rootfs - #629

Open
stubbi wants to merge 47 commits into
mainfrom
fix/netbird-readonly-rootfs
Open

stubbi wants to merge 47 commits into
mainfrom
fix/netbird-readonly-rootfs

Conversation

@stubbi

@stubbi stubbi commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Problem

The spec.netbird sidecar crash-loops with the standard netbirdio/netbird image (#624). The image entrypoint starts netbird service run before netbird up, and both default to paths on the root filesystem, which the operator makes read-only:

Failed to write to log, can't make directories for new logfile: mkdir /var/log/netbird: read-only file system
Error: listen daemon interface: listen unix /var/run/netbird.sock: bind: read-only file system
daemon failed to start, exiting...

Change

In internal/resources/mesh_netbird.go the sidecar now also gets:

  • NB_DAEMON_ADDR=unix:///tmp/netbird/netbird.sock: the daemon socket lives on the netbird-state emptyDir. Both service run and up read it (NetBird maps every persistent flag to NB_<FLAG>, here --daemon-addr).
  • NB_LOG_FILE=console,/tmp/netbird/client.log: logs still go to the console. A real file is kept as well because the entrypoint tails it for started daemon server before it runs netbird up. With console only, it would sleep blindly instead. The client rotates the file itself (lumberjack).
  • USER=netbird: once the first two were fixed, netbird up failed next with user: Current requires cgo or $USER set in environment, because the pod UID has no passwd entry in the image.

No API change and no new volume. The image entrypoint is unchanged.

Testing

  • Reproduced and verified locally with netbirdio/netbird:0.60.8 under docker run --read-only --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges, a writable tmpfs at /tmp/netbird, and the operator's env:
    • before: the two errors above, exit 1 after 5 s.
    • after: started daemon server: /tmp/netbird/netbird.sock, then netbird up gets as far as contacting the management URL. With an unreachable loopback URL and a fake key it then fails on enrollment, which is the expected end of this test, the same scope as the report.
  • New unit test TestNetBirdSidecar_WritablePathsLiveOnMountedVolume: the daemon socket, log file, state dir and config must all sit on a writable mount, and USER must be set.
  • New e2e spec Should start the NetBird daemon under a read-only root filesystem: starts the real image in kind and waits for started daemon server in the sidecar log, with no read-only file system error. This is the container startup test the report asked for.
  • I could not run make test locally (the machine ran out of disk), so unit tests, lint and e2e run in CI only.

Not covered

  • Inbound forwarding (NB_ENABLE_NETSTACK_LOCAL_FORWARDING) and real enrollment are untested, as noted in the issue.

Fixes #624

🤖 Generated with Claude Code

stubbi and others added 23 commits October 3, 2026 18:59
With spec.storage.fixOwnership unset, the controller reads the instance's
Namespace to decide whether init-data-owner is admissible. When that read
failed (for example RBAC without the namespaces "get" rule), it kept the
init container enabled, which could flip a working StatefulSet in a
restricted namespace to a Pod template admission rejects.

The read failure now aborts reconciliation before any managed resource is
touched, surfaces through the ReconcileFailed event and Ready condition,
and is retried. An explicit fixOwnership still bypasses the lookup.

Adds envtest coverage: a restricted-compatible instance whose namespace
read then fails keeps its StatefulSet unchanged, and explicit values skip
the lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…otfs

The NetBird image entrypoint starts "netbird service run" before
"netbird up". Both default the daemon socket to
unix:///var/run/netbird.sock and the log file to
/var/log/netbird/client.log, which cannot be created on the read-only
root filesystem the operator gives the sidecar, so the daemon exited
after five seconds and the container crash-looped before enrolling.

Point NB_DAEMON_ADDR and NB_LOG_FILE at the netbird-state emptyDir. A
real log file is kept (with console output) because the entrypoint tails
it to detect daemon startup. Also set USER: the pod UID has no passwd
entry and the cgo-less CLI cannot resolve the current user without it,
which made "netbird up" fail next.

Reproduced and verified with netbirdio/netbird:0.60.8 under docker run
--read-only --user 1000:1000 --cap-drop ALL: before, the daemon fails on
mkdir /var/log/netbird and bind /var/run/netbird.sock; after, it logs
"started daemon server: /tmp/netbird/netbird.sock" and proceeds to
enrollment.

Fixes #624

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stubbi
stubbi force-pushed the fix/netbird-readonly-rootfs branch from 9c28585 to c1a2e1b Compare October 11, 2026 16:55
jtatum and others added 4 commits October 11, 2026 12:55
In `watchNamespaces` mode, #618 places `get namespaces` in tenant Roles.
Namespace objects are cluster-scoped, so those grants cannot authorize
the Pod Security label lookup. The controller then retains the ownership
helper default, potentially producing a pod rejected by restricted
admission.

Move Namespace reads into the chart's existing cluster-scoped rule set
and ClusterRoleBinding. In namespaced mode, constrain `resourceNames` to
the configured watched namespaces. Cluster-wide mode permits reading any
Namespace. Neither mode gains Namespace list/watch/write permissions.

This PR targets `fix/data-owner-pss-aware` as a focused follow-up to
#618. It can be retargeted/rebased onto main after that PR merges. The
separate namespace-lookup failure handling review on #618 remains
relevant; this change fixes the chart permission grant.

Validation:
- A regression test renders the chart, applies its actual RBAC to a
disposable envtest API server configured with RBAC, and checks
authorization as the rendered operator ServiceAccount identity.
- The original #618 template fails the watched-Namespace read assertion.
The fixed template permits the two watched namespaces, denies
unwatched/operator namespaces, and denies Namespace list/watch/patch.
Cluster-wide mode is checked too.
- Full `make test`, `make lint`, and both Helm RBAC guard scripts pass.
CI installs Helm for the authorization test.

---------

Co-authored-by: Jannes Stubbemann <jannes.stubbemann@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…en enabled

Addresses review on #617:

- Pin mcp.apps.sandboxPort to 18795 with the gateway proxy disabled too,
  and make the Service, NetworkPolicy, container port and nginx upstream
  follow the effective listener port (a user-supplied port wins). With the
  proxy disabled the Service used to target 18795 while OpenClaw listened
  on its default 18790.
- Only render the sandbox container ports, nginx listener, Service port
  and NetworkPolicy ingress when spec.config.raw enables MCP Apps, so
  instances that do not use it see no change. With configMapRef or
  mergeMode=merge the operator cannot see the effective config and keeps
  exposing the sandbox.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
init-uv mounts the data volume at /data, while the gateway sees it at
/home/openclaw/.openclaw and /home/openclaw/.local. An absolute
~/.local/bin/uv symlink created from the gateway therefore dangles in
init-uv: the -x check fails, cp refuses to write through the dangling
link, and with set -e the pod stays in Init:CrashLoopBackOff until the
link is removed from the volume by hand.

The script now maps the two gateway mount points back to /data before
inspecting a symlink. A link whose target exists is kept, so a pinned uv
survives restarts. A link whose target is gone is removed and replaced.
The binary is copied to a temporary file and renamed over the
destination, so the script never writes through a symlink.

Fixes #626

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stubbi and others added 15 commits October 11, 2026 12:56
# Conflicts:
#	api/v1alpha1/openclawinstance_types.go
#	bundle/manifests/openclaw.rocks_openclawinstances.yaml
#	charts/openclaw-operator/templates/crds/openclaw.rocks_openclawinstances.yaml
#	config/crd/bases/openclaw.rocks_openclawinstances.yaml
#	docs/api-reference.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#619 made every operator init container that writes pod storage use the
gateway's effective UID. Do the same for init-tmp-dir so the sticky /tmp
directory belongs to the user that writes to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release Please has failed on every push since the repository moved from
paperclipinc to stubbi: vars.RELEASE_BOT_APP_ID and
secrets.RELEASE_BOT_PRIVATE_KEY were settings of the old organization, so
actions/create-github-app-token aborts with "Input required and not
supplied: app-id" before release-please runs.

- release-please.yaml and release.yaml mint the App token only when the
  App is configured, then fall back to RELEASE_PLEASE_TOKEN and finally
  GITHUB_TOKEN. With GITHUB_TOKEN, tag pushes and release publishing do
  not trigger other workflows, so release.yaml and operatorhub.yaml are
  dispatched explicitly (release.yaml gains a workflow_dispatch trigger
  and only runs on tag refs).
- The GHCR namespace for the image and the chart comes from one place,
  vars.IMAGE_NAMESPACE (default paperclipinc), used by release.yaml and
  .goreleaser.yaml. release.yaml previously used the repository owner
  (now stubbi) for signing and the chart while GoReleaser pushed to
  paperclipinc. A preflight fails the release if the namespace does not
  match the chart's default image.repository (#536).
- GHCR login accepts GHCR_PUSH_TOKEN / GHCR_PUSH_USER, needed to push
  into a namespace the repository owner does not own.
- GoReleaser creates the release in stubbi/openclaw-operator.
- make verify-signing accepts both the old and the new workflow identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stubbi

stubbi commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Merge order note (maintenance pass, 2026-10-11). The open stubbi PRs are stacked so they apply cleanly in this order: #623 → #627 → #620 → #618 → #617 → #628 → #629. This is position 7 of 7. This branch has #628 (and everything below it) merged in, so until those land the diff here also shows their changes; once they are squash-merged, "Update branch" reduces it to this PR's own changes without conflicts. #623 is at the bottom because the required E2E check cannot pass without its sticky /tmp fix (OpenClaw >= 2026.9.6 crash-loops otherwise). Branch protection requires an up-to-date branch, so after each merge click "Update branch" on the next PR and let CI re-run before merging it.

stubbi and others added 4 commits October 11, 2026 14:13
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new e2e startup spec showed two more writes to the read-only root:
"netbird up" tried to mkdir /.config because the pod UID has no passwd
entry (HOME falls back to /), and the daemon's updater removes a
result.json relative to its working directory (/). Set HOME and
workingDir to the writable state volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please push updates or leave a comment.

@github-actions github-actions Bot added the stale label Oct 19, 2026
@jtatum

jtatum commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Not stale

@github-actions github-actions Bot removed the stale label Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: NetBird sidecar fails to start with read-only rootfs (log and daemon socket paths)

2 participants