Skip to content

feat(netguard): confine untrusted code egress to the public internet - #520

Merged
taubyte0 merged 8 commits into
mainfrom
fix/network-egress-isolation
Aug 16, 2026
Merged

taubyte0 merged 8 commits into
mainfrom
fix/network-egress-isolation

Conversation

@samyfodil

Copy link
Copy Markdown
Contributor

Untrusted code on a node — a tenant WASM function, a container running a
repository's own build.sh, and a repository's own Dockerfile — could reach
node-local services and the cloud metadata endpoint (169.254.169.254). This adds
one egress policy, pkg/netguard, and enforces it at two layers.

Layer A — guest (pure Go, every OS). A WASM function's only outbound network
is through host functions we own, so it is filtered in-process: a denied-CIDR set
plus stdlib flag checks, a net.Dialer.Control guard that runs after DNS
resolution (so it sees the resolved address, not the hostname, defeating
rebinding), and a redirect guard on the guest HTTP client. The client also got a
timeout and the host-call context threaded through it, so a hung request can no
longer pin the pooled instance.

Layer B — container (Linux, nftables, fail-closed). build.sh shells out to
tools that never see our client, so it is filtered at the host firewall instead.
Restricted containers get a network and a cgroup of their own, and both backends
install the rules before the container can be created:

  • docker — an inet taubyte_netguard table dropping denied destinations from
    tau-netguard0, the pinned bridge of a network reserved for restricted
    containers. Nothing else on the host is filtered.
  • containerd (host netns, no CNI) — restricted containers are pinned under
    /<namespace>/restricted/<id> and one socket cgroupv2 rule matched to that
    parent covers them all, nested cgroups included. Rootless and non-cgroup-v2
    hosts are refused rather than run unfiltered.

A repository's Dockerfile is untrusted for the same reason its build.sh is,
and its RUN steps run first. The builder is restricted like any other untrusted
workload, and BuildKit is given a cgroup parent so the runc containers it spawns
per step land inside the filtered subtree instead of at the root of the host
hierarchy.

The table is programmed over netlink in-process, so the node needs
CAP_NET_ADMIN; without it the flush returns EPERM and the container is not
created (the job re-dispatches elsewhere). It is built in a single atomic
transaction — no ruleless window — and re-asserted per container, so an
out-of-band flush is repaired.

Restriction applies only off a development node and only on Linux: enforcement
is a host firewall, so on any other OS there is nothing to program, and a laptop
or a dream universe has neither the privileges nor an infrastructure behind it.
The builder makes that call, with monkey passing its own dev flag in.

Notes on the parts that only a kernel could settle

  • The kernel rejects the nftables library's Key/KeyEnd interval-set form with
    EINVAL. Interval sets are boundary pairs — a start element and an exclusive end
    marker — which is what nft itself emits. Ranges are sorted and merged first,
    because two adjacent ones would otherwise emit an end marker and a start
    element carrying the same key.
  • A restricted container inherited the host's resolv.conf, whose nameserver on
    a systemd host is the resolved stub at 127.0.0.53 — loopback, and therefore
    denied. Every restricted container silently lost name resolution. It now gets
    the host's nameservers minus the ones the policy denies, falling back to
    systemd's upstreams and then to public resolvers.
  • CAP_NET_RAW opens AF_PACKET, which writes straight to the device and never
    passes the hook the rule lives in. Dropped for restricted containers, which
    also denies it to anything they spawn. Not for privileged ones: BuildKit needs
    it and CAP_NET_ADMIN both, so a Dockerfile's RUN steps are confined by the
    cgroup rule but keep AF_PACKET. Giving those steps a network namespace of their
    own is what would close that, and it is a change to how BuildKit is run.
  • Nothing matched an address the host itself holds, so on a node with a routable
    address every service bound to 0.0.0.0 stayed reachable. Both chains now ask
    the routing table for the destination type.
  • The nested-cgroup-namespace guard could not detect its own case: a process in
    one reports the same 0::/ as a process at the real root. It checks the mount
    instead.

Tests

Unit tests cover the policy, the encoding, and the merge, and one asserts the two
layers cannot drift — every address IsDenied rejects must be covered by a
prefix the firewall is built from.

Two build-tagged suites program a real firewall and are wired into CI under sudo,
without which none of it is exercised anywhere, since the unprivileged pass skips
it all. They assert what the code claims: a restricted container is refused a
node-local service — on loopback, and on an address of the host outside every
denied CIDR — still reaches the public internet, still resolves names, holds no
CAP_NET_RAW, and leaves unrestricted containers alone; and a Dockerfile RUN
step is refused the same service while the build still reaches a registry.

Verified on kernel 6.8 in the rootful containerd VM, and the full containerd
conformance suite still passes unprivileged.

Untrusted code on a node — a tenant WASM function or a build container running
repo-supplied build.sh — could reach node-local services and the cloud metadata
endpoint (169.254.169.254), an SSRF into the infrastructure. This adds one
egress policy (pkg/netguard) enforced at two layers.

Layer A — guest (pure Go, all OSes). A WASM function's only outbound network is
through host functions we own, so it is filtered in-process:
- pkg/netguard: denied CIDR set + stdlib flag checks, IsDenied, a
  net.Dialer.Control guard that runs after DNS resolution (defeats rebinding),
  and RestrictedDialer.
- pkg/vm-low-orbit/http/client: the guest HTTP client dials through the guard,
  with a timeout and the host-call ctx threaded in (a hung request can no longer
  pin the pooled instance) and a redirect guard.
- pkg/vm-low-orbit/dns: the tenant-controllable resolver (reroute and reset)
  dials through the same guard.

Layer B — build container (Linux, nftables, fail-closed). build.sh shells out to
tools that bypass our client, so it is filtered at the host firewall. Every build
container is marked RestrictedEgress() via DefaultOptions, and both backends
enforce it before the build can run:
- docker: an nftables table (inet taubyte_netguard) dropping denied destinations
  from the docker0 bridge (forward + input), installed atomically; host and
  custom network modes are rejected. Installed before ContainerCreate.
- containerd (host netns, no CNI): build containers are pinned under the
  namespace cgroup and a single socket-cgroupv2 output rule matched to that
  parent covers them all (including BuildKit's nested runc cgroups), installed
  before the container is created. Rootless and non-cgroup-v2 hosts are refused.

Firewall programming uses github.com/google/nftables over netlink in-process, so
the node needs CAP_NET_ADMIN; without it the flush returns EPERM and the build
fails closed (the job re-dispatches to another node). The nftables table is
built in a single atomic transaction (no ruleless window) and re-asserted per
build (idempotent), so an out-of-band flush is repaired.

The nftables/cgroup code is written against the real google/nftables API and
unit-tested for rule construction, but the socket-cgroupv2 match semantics and
actual rule application require a CAP_NET_ADMIN Linux host (kernel >= 5.13) to
validate end-to-end — see the VALIDATION notes in the firewall files.
… a kernel

The policy and both installers were written against the nftables API but never
run against a kernel. Doing that changed several of them, and turned up two
holes and a functional break.

Encoding. The kernel rejects the library's Key/KeyEnd interval-set form with
EINVAL; interval sets are boundary pairs — a start element and an exclusive end
marker — which is what nft itself emits. Ranges are now sorted and merged before
encoding, because two adjacent ranges (::/128 and ::1/128) would otherwise emit
an end marker and a start element carrying the same key, and a range ending at
the last address of its family emits no marker at all.

Scope. Both installers filtered every container on the host, not the restricted
ones: the docker rule matched the shared docker0, and the containerd rule matched
the whole namespace cgroup, which is also every unrestricted container's parent.
Restricted containers now get a network of their own (taubyte_netguard, bridge
pinned to tau-netguard0) and a cgroup of their own (/<ns>/restricted/<id>,
matched at level 2). Neither backend touches anything else on the machine.

DNS. A restricted container inherited the host's resolv.conf, whose nameserver
on a systemd host is the resolved stub at 127.0.0.53 — loopback, and therefore
denied. Every restricted container silently lost name resolution. It now gets
the host's nameservers minus the ones the policy denies, falling back to
systemd's upstreams and then to public resolvers. The same guard had been added
to the guest's dns.Reset(), where it broke every lookup for the same reason and
blocked no attack — the guest cannot aim that resolver — so Reset() is back to
the host's own. Reroute keeps its guard: there the address is guest-chosen.

Holes found by review and closed:

- A repository's Dockerfile was built with no restriction at all. The builder is
  now restricted like any other untrusted workload, and BuildKit is given a
  cgroup parent so the runc containers it spawns per RUN step land inside the
  filtered subtree instead of at the root of the host hierarchy.
- CAP_NET_RAW opens AF_PACKET, which writes straight to the device and never
  passes the hook the rule lives in. Dropped for restricted containers, which
  also denies it to anything they spawn. Not for privileged ones: BuildKit needs
  both it and CAP_NET_ADMIN, so a Dockerfile's RUN steps are confined by the
  cgroup rule but keep AF_PACKET.
- Nothing matched an address the host itself holds, so on a node with a routable
  address every service bound to 0.0.0.0 stayed reachable. Both chains now ask
  the routing table for the destination type.
- The two layers had already drifted: IsDenied rejected multicast, the firewall
  did not. Multicast, 240/4 and Azure's WireServer joined the policy, and a test
  now asserts every address IsDenied rejects is covered by a prefix.
- The nested-cgroup-namespace guard could not detect its own case: a process in
  one reports the same "0::/" as a process at the real root. It checks the mount
  instead. Docker refuses when its bridge is not in tau's network namespace,
  which is what a rootless daemon does.

Tests. Two build-tagged integration suites that program a real firewall, plus
the CI jobs to run them under sudo — without which none of this is exercised
anywhere, since the unprivileged pass skips it all. They assert what the code
claims: a restricted container is refused a node-local service (on loopback and
on an address of the host outside every denied CIDR), still reaches the public
internet, still resolves names, holds no CAP_NET_RAW, and leaves unrestricted
containers alone; and a Dockerfile RUN step is refused the same service while
the build still reaches a registry.
…rict it with

Enforcement is a host firewall — nftables plus cgroups — so off Linux there is
nothing to program, and on a development node there are neither the privileges
to program one nor an infrastructure behind it worth reaching. Asking anyway
failed every build closed on a laptop, in a dream universe, and under `tau
build`, which is how the CLI, the builder's own tests and the tau e2e tests
started failing.

The decision moves to the one caller that can make it. DefaultOptions no longer
attaches RestrictedEgress: the builder does, when the node is not in dev mode
and the platform is Linux, and monkey passes its own dev flag in. A Dockerfile
build takes the same decision the same way rather than assuming it, so the
backends now honour what they are asked for instead of restricting whatever
comes through the image path.

That also closes the docker half of the same hole: a repository's Dockerfile
built through the docker backend ran its RUN steps on the default bridge, which
the rule does not match. It now runs on tau's network, fails closed like Create
when that network is not one this process can filter.
The fixture's monkey embeds a nil Service and overrides only what jobs used to
reach for. Passing the node's dev flag into the builder made the job reach for
one more, and the call landed on nothing.
… does

Tarball skips anything that is not a regular file (utils/bundle/tarball.go);
the zip side did not, and read through whatever a name resolved to. The two now
agree, and the builder resolves an output path before handing it over, keeping
it under the directory the build wrote.

Adds the package's first tests.
…n out of a restricted build's network

Two gaps in what the egress policy actually covers.

The backend manager builds a core.ContainerdConfig literal with no namespace
(backend_manager.go), and nothing filled one in. containerd requires a namespace
on every call, and the restricted cgroup is named after it, so a container asking
for restricted egress failed closed on any node that took that path. The backend
defaults it now.

ADD is the one instruction the docker daemon carries out itself, in its own
network namespace rather than in a container, so the bridge rule cannot cover it
and the destination is whatever the Dockerfile names. A restricted build that
uses a remote ADD is refused, and pointed at COPY.
…tree, rootless included

Restricted egress used to require a rootful containerd, because it needed a
cgroup to name and a rootless daemon was given none (withoutCgroups). That is
backwards: rootless is the mode with no privileged container anywhere in it, and
the one where a workload cannot reach a network device at all — its traffic
leaves through a userspace stack. Refusing it left the weaker configuration as
the only one that could be confined.

tau now takes a subtree of its own — the cgroup systemd delegated to its unit, or
one it creates when running as root — and puts everything under it:

  <tau>/main            tau itself, so the parent holds no processes and can
                        hand its controllers down (the kernel allows one or the
                        other, not both)
  <tau>/restricted      what the egress rule covers, matched by depth so
                        everything below is included
  <tau>/restricted/daemon   the rootless daemon, in a leaf of its own for the
                        same no-internal-process reason
  <tau>/containers      containers that asked for nothing, deliberately beside
                        restricted rather than under it

The rootless daemon starting inside restricted is what makes the mode work at
all: a rootless container's traffic is forwarded by a userspace network stack in
that process, so the sockets the host sees belong to it, and the rule finds them
by that cgroup or not at all.

Resource limits come back with it. They were refused under rootless for the same
reason egress was, and they are the answer to a build that forks until the node
falls over. They need the controllers delegated, so they are offered when that
worked and refused explicitly when it did not, rather than assumed.

The level the rule matches is now the restricted cgroup's real depth rather than
one derived from the containerd namespace, which is what lets the subtree sit
wherever systemd put tau.

Adds an integration test that runs the rootless path end to end: a workload in
the restricted cgroup is refused a node-local service and still reaches the
public internet.
…ork a ceiling

The host kernel is the only thing standing between a container and the node when
the runtime is runc, and every hardening step so far — dropping capabilities,
matching cgroups, filtering egress — works around that rather than changing it.
containerd already lets a client name a different runtime, so a node that runs
untrusted code can name a sandboxed one and get a boundary instead of a set of
mitigations.

Runtime is a handler name in the backend's config, applied to restricted
containers only. Empty keeps containerd's default. Naming it rather than
building a flag around one implementation is deliberate: what is worth running
today may not be what is worth running next year, and swapping it should be
configuration.

Also here, two things a restricted container should have had already:

A process cap. A build that forks until the node falls over needs no exploit and
no network, and it takes the node's other tenants with it. 4096 by default,
overridden by a caller that asks for its own, and only where the cgroup subtree
can carry it. Memory is left alone on purpose — a guessed limit fails real
builds and belongs in configuration.

A build context that unpacks. Builders gzip the tarball (utils/bundle) while
tests and API callers usually do not; the docker daemon sniffs the difference
and this backend did not, so a repository's Dockerfile could never have built
here at all.
@taubyte0
taubyte0 merged commit d984166 into main Aug 16, 2026
24 checks passed
@taubyte0
taubyte0 deleted the fix/network-egress-isolation branch August 16, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants