feat(netguard): confine untrusted code egress to the public internet - #520
Merged
Merged
Conversation
Untrusted code on a node — a tenant WASM function or a build container running repo-supplied build.sh — could reach node-local services and the cloud metadata endpoint (169.254.169.254), an SSRF into the infrastructure. This adds one egress policy (pkg/netguard) enforced at two layers. Layer A — guest (pure Go, all OSes). A WASM function's only outbound network is through host functions we own, so it is filtered in-process: - pkg/netguard: denied CIDR set + stdlib flag checks, IsDenied, a net.Dialer.Control guard that runs after DNS resolution (defeats rebinding), and RestrictedDialer. - pkg/vm-low-orbit/http/client: the guest HTTP client dials through the guard, with a timeout and the host-call ctx threaded in (a hung request can no longer pin the pooled instance) and a redirect guard. - pkg/vm-low-orbit/dns: the tenant-controllable resolver (reroute and reset) dials through the same guard. Layer B — build container (Linux, nftables, fail-closed). build.sh shells out to tools that bypass our client, so it is filtered at the host firewall. Every build container is marked RestrictedEgress() via DefaultOptions, and both backends enforce it before the build can run: - docker: an nftables table (inet taubyte_netguard) dropping denied destinations from the docker0 bridge (forward + input), installed atomically; host and custom network modes are rejected. Installed before ContainerCreate. - containerd (host netns, no CNI): build containers are pinned under the namespace cgroup and a single socket-cgroupv2 output rule matched to that parent covers them all (including BuildKit's nested runc cgroups), installed before the container is created. Rootless and non-cgroup-v2 hosts are refused. Firewall programming uses github.com/google/nftables over netlink in-process, so the node needs CAP_NET_ADMIN; without it the flush returns EPERM and the build fails closed (the job re-dispatches to another node). The nftables table is built in a single atomic transaction (no ruleless window) and re-asserted per build (idempotent), so an out-of-band flush is repaired. The nftables/cgroup code is written against the real google/nftables API and unit-tested for rule construction, but the socket-cgroupv2 match semantics and actual rule application require a CAP_NET_ADMIN Linux host (kernel >= 5.13) to validate end-to-end — see the VALIDATION notes in the firewall files.
… a kernel The policy and both installers were written against the nftables API but never run against a kernel. Doing that changed several of them, and turned up two holes and a functional break. Encoding. The kernel rejects the library's Key/KeyEnd interval-set form with EINVAL; interval sets are boundary pairs — a start element and an exclusive end marker — which is what nft itself emits. Ranges are now sorted and merged before encoding, because two adjacent ranges (::/128 and ::1/128) would otherwise emit an end marker and a start element carrying the same key, and a range ending at the last address of its family emits no marker at all. Scope. Both installers filtered every container on the host, not the restricted ones: the docker rule matched the shared docker0, and the containerd rule matched the whole namespace cgroup, which is also every unrestricted container's parent. Restricted containers now get a network of their own (taubyte_netguard, bridge pinned to tau-netguard0) and a cgroup of their own (/<ns>/restricted/<id>, matched at level 2). Neither backend touches anything else on the machine. DNS. A restricted container inherited the host's resolv.conf, whose nameserver on a systemd host is the resolved stub at 127.0.0.53 — loopback, and therefore denied. Every restricted container silently lost name resolution. It now gets the host's nameservers minus the ones the policy denies, falling back to systemd's upstreams and then to public resolvers. The same guard had been added to the guest's dns.Reset(), where it broke every lookup for the same reason and blocked no attack — the guest cannot aim that resolver — so Reset() is back to the host's own. Reroute keeps its guard: there the address is guest-chosen. Holes found by review and closed: - A repository's Dockerfile was built with no restriction at all. The builder is now restricted like any other untrusted workload, and BuildKit is given a cgroup parent so the runc containers it spawns per RUN step land inside the filtered subtree instead of at the root of the host hierarchy. - CAP_NET_RAW opens AF_PACKET, which writes straight to the device and never passes the hook the rule lives in. Dropped for restricted containers, which also denies it to anything they spawn. Not for privileged ones: BuildKit needs both it and CAP_NET_ADMIN, so a Dockerfile's RUN steps are confined by the cgroup rule but keep AF_PACKET. - Nothing matched an address the host itself holds, so on a node with a routable address every service bound to 0.0.0.0 stayed reachable. Both chains now ask the routing table for the destination type. - The two layers had already drifted: IsDenied rejected multicast, the firewall did not. Multicast, 240/4 and Azure's WireServer joined the policy, and a test now asserts every address IsDenied rejects is covered by a prefix. - The nested-cgroup-namespace guard could not detect its own case: a process in one reports the same "0::/" as a process at the real root. It checks the mount instead. Docker refuses when its bridge is not in tau's network namespace, which is what a rootless daemon does. Tests. Two build-tagged integration suites that program a real firewall, plus the CI jobs to run them under sudo — without which none of this is exercised anywhere, since the unprivileged pass skips it all. They assert what the code claims: a restricted container is refused a node-local service (on loopback and on an address of the host outside every denied CIDR), still reaches the public internet, still resolves names, holds no CAP_NET_RAW, and leaves unrestricted containers alone; and a Dockerfile RUN step is refused the same service while the build still reaches a registry.
…rict it with Enforcement is a host firewall — nftables plus cgroups — so off Linux there is nothing to program, and on a development node there are neither the privileges to program one nor an infrastructure behind it worth reaching. Asking anyway failed every build closed on a laptop, in a dream universe, and under `tau build`, which is how the CLI, the builder's own tests and the tau e2e tests started failing. The decision moves to the one caller that can make it. DefaultOptions no longer attaches RestrictedEgress: the builder does, when the node is not in dev mode and the platform is Linux, and monkey passes its own dev flag in. A Dockerfile build takes the same decision the same way rather than assuming it, so the backends now honour what they are asked for instead of restricting whatever comes through the image path. That also closes the docker half of the same hole: a repository's Dockerfile built through the docker backend ran its RUN steps on the default bridge, which the rule does not match. It now runs on tau's network, fails closed like Create when that network is not one this process can filter.
The fixture's monkey embeds a nil Service and overrides only what jobs used to reach for. Passing the node's dev flag into the builder made the job reach for one more, and the call landed on nothing.
… does Tarball skips anything that is not a regular file (utils/bundle/tarball.go); the zip side did not, and read through whatever a name resolved to. The two now agree, and the builder resolves an output path before handing it over, keeping it under the directory the build wrote. Adds the package's first tests.
…n out of a restricted build's network Two gaps in what the egress policy actually covers. The backend manager builds a core.ContainerdConfig literal with no namespace (backend_manager.go), and nothing filled one in. containerd requires a namespace on every call, and the restricted cgroup is named after it, so a container asking for restricted egress failed closed on any node that took that path. The backend defaults it now. ADD is the one instruction the docker daemon carries out itself, in its own network namespace rather than in a container, so the bridge rule cannot cover it and the destination is whatever the Dockerfile names. A restricted build that uses a remote ADD is refused, and pointed at COPY.
…tree, rootless included
Restricted egress used to require a rootful containerd, because it needed a
cgroup to name and a rootless daemon was given none (withoutCgroups). That is
backwards: rootless is the mode with no privileged container anywhere in it, and
the one where a workload cannot reach a network device at all — its traffic
leaves through a userspace stack. Refusing it left the weaker configuration as
the only one that could be confined.
tau now takes a subtree of its own — the cgroup systemd delegated to its unit, or
one it creates when running as root — and puts everything under it:
<tau>/main tau itself, so the parent holds no processes and can
hand its controllers down (the kernel allows one or the
other, not both)
<tau>/restricted what the egress rule covers, matched by depth so
everything below is included
<tau>/restricted/daemon the rootless daemon, in a leaf of its own for the
same no-internal-process reason
<tau>/containers containers that asked for nothing, deliberately beside
restricted rather than under it
The rootless daemon starting inside restricted is what makes the mode work at
all: a rootless container's traffic is forwarded by a userspace network stack in
that process, so the sockets the host sees belong to it, and the rule finds them
by that cgroup or not at all.
Resource limits come back with it. They were refused under rootless for the same
reason egress was, and they are the answer to a build that forks until the node
falls over. They need the controllers delegated, so they are offered when that
worked and refused explicitly when it did not, rather than assumed.
The level the rule matches is now the restricted cgroup's real depth rather than
one derived from the containerd namespace, which is what lets the subtree sit
wherever systemd put tau.
Adds an integration test that runs the rootless path end to end: a workload in
the restricted cgroup is refused a node-local service and still reaches the
public internet.
…ork a ceiling The host kernel is the only thing standing between a container and the node when the runtime is runc, and every hardening step so far — dropping capabilities, matching cgroups, filtering egress — works around that rather than changing it. containerd already lets a client name a different runtime, so a node that runs untrusted code can name a sandboxed one and get a boundary instead of a set of mitigations. Runtime is a handler name in the backend's config, applied to restricted containers only. Empty keeps containerd's default. Naming it rather than building a flag around one implementation is deliberate: what is worth running today may not be what is worth running next year, and swapping it should be configuration. Also here, two things a restricted container should have had already: A process cap. A build that forks until the node falls over needs no exploit and no network, and it takes the node's other tenants with it. 4096 by default, overridden by a caller that asks for its own, and only where the cgroup subtree can carry it. Memory is left alone on purpose — a guessed limit fails real builds and belongs in configuration. A build context that unpacks. Builders gzip the tarball (utils/bundle) while tests and API callers usually do not; the docker daemon sniffs the difference and this backend did not, so a repository's Dockerfile could never have built here at all.
taubyte0
approved these changes
Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Untrusted code on a node — a tenant WASM function, a container running a
repository's own
build.sh, and a repository's ownDockerfile— could reachnode-local services and the cloud metadata endpoint (169.254.169.254). This adds
one egress policy,
pkg/netguard, and enforces it at two layers.Layer A — guest (pure Go, every OS). A WASM function's only outbound network
is through host functions we own, so it is filtered in-process: a denied-CIDR set
plus stdlib flag checks, a
net.Dialer.Controlguard that runs after DNSresolution (so it sees the resolved address, not the hostname, defeating
rebinding), and a redirect guard on the guest HTTP client. The client also got a
timeout and the host-call context threaded through it, so a hung request can no
longer pin the pooled instance.
Layer B — container (Linux, nftables, fail-closed).
build.shshells out totools that never see our client, so it is filtered at the host firewall instead.
Restricted containers get a network and a cgroup of their own, and both backends
install the rules before the container can be created:
inet taubyte_netguardtable dropping denied destinations fromtau-netguard0, the pinned bridge of a network reserved for restrictedcontainers. Nothing else on the host is filtered.
/<namespace>/restricted/<id>and onesocket cgroupv2rule matched to thatparent covers them all, nested cgroups included. Rootless and non-cgroup-v2
hosts are refused rather than run unfiltered.
A repository's
Dockerfileis untrusted for the same reason itsbuild.shis,and its
RUNsteps run first. The builder is restricted like any other untrustedworkload, and BuildKit is given a cgroup parent so the runc containers it spawns
per step land inside the filtered subtree instead of at the root of the host
hierarchy.
The table is programmed over netlink in-process, so the node needs
CAP_NET_ADMIN; without it the flush returns EPERM and the container is notcreated (the job re-dispatches elsewhere). It is built in a single atomic
transaction — no ruleless window — and re-asserted per container, so an
out-of-band flush is repaired.
Restriction applies only off a development node and only on Linux: enforcement
is a host firewall, so on any other OS there is nothing to program, and a laptop
or a dream universe has neither the privileges nor an infrastructure behind it.
The builder makes that call, with monkey passing its own dev flag in.
Notes on the parts that only a kernel could settle
Key/KeyEndinterval-set form withEINVAL. Interval sets are boundary pairs — a start element and an exclusive end
marker — which is what
nftitself emits. Ranges are sorted and merged first,because two adjacent ones would otherwise emit an end marker and a start
element carrying the same key.
resolv.conf, whose nameserver ona systemd host is the resolved stub at 127.0.0.53 — loopback, and therefore
denied. Every restricted container silently lost name resolution. It now gets
the host's nameservers minus the ones the policy denies, falling back to
systemd's upstreams and then to public resolvers.
CAP_NET_RAWopens AF_PACKET, which writes straight to the device and neverpasses the hook the rule lives in. Dropped for restricted containers, which
also denies it to anything they spawn. Not for privileged ones: BuildKit needs
it and
CAP_NET_ADMINboth, so aDockerfile'sRUNsteps are confined by thecgroup rule but keep AF_PACKET. Giving those steps a network namespace of their
own is what would close that, and it is a change to how BuildKit is run.
address every service bound to
0.0.0.0stayed reachable. Both chains now askthe routing table for the destination type.
one reports the same
0::/as a process at the real root. It checks the mountinstead.
Tests
Unit tests cover the policy, the encoding, and the merge, and one asserts the two
layers cannot drift — every address
IsDeniedrejects must be covered by aprefix the firewall is built from.
Two build-tagged suites program a real firewall and are wired into CI under sudo,
without which none of it is exercised anywhere, since the unprivileged pass skips
it all. They assert what the code claims: a restricted container is refused a
node-local service — on loopback, and on an address of the host outside every
denied CIDR — still reaches the public internet, still resolves names, holds no
CAP_NET_RAW, and leaves unrestricted containers alone; and aDockerfileRUNstep is refused the same service while the build still reaches a registry.
Verified on kernel 6.8 in the rootful containerd VM, and the full containerd
conformance suite still passes unprivileged.