Skip to content

Networks: the address pool skips only on-link routes, as Docker does (#228) - #351

Open
sundarshahi wants to merge 1 commit into
libracoder:mainfrom
sundarshahi:fix/228-pool-ignores-gateway-routes
Open

sundarshahi wants to merge 1 commit into
libracoder:mainfrom
sundarshahi:fix/228-pool-ignores-gateway-routes

Conversation

@sundarshahi

Copy link
Copy Markdown

What this changes and why

On a Linux host, docker network create with no --subnet can fail with no free subnet left even when 172.18–172.31 are all free. kind runs exactly that command, so kind create cluster fails too. Docker Engine creates the network on the same host.

Kiri keeps every subnet a host route covers out of its address pool. Since 23.0, Docker keeps out only on-link routes (moby#42598, queryOnLinkRoutes in daemon/libnetwork/netutils/utils_linux.go). A route through a gateway doesn't mean the subnet is in use. A full-tunnel VPN shows why that matters: OpenVPN's def1 adds 0.0.0.0/1 and 128.0.0.0/1 through the tunnel's gateway, and those two routes cover every subnet Kiri would pick.

Refs #228. I could reproduce the error, but the report doesn't include the host's routes. @MAST1999, could you post cat /proc/net/route from the machine where it fails? That would show whether this is your cause.

Problem

  • free_subnet (crates/kiri/src/daemon/network.rs:746) skips any candidate that overlaps host_routes().
  • host_routes (:787) kept every non-default route from /proc/net/route, including:
    • routes through a gateway (flag G, e.g. 0.0.0.0/1 via 10.8.0.1 dev tun0);
    • blackhole and unreachable routes (interface *).
  • With the two /1 routes, every 172.x/16 and 10.89.x/24 candidate overlaps, and create returns no free subnet left (:622).

Fix

host_routes now keeps only the routes that put the host on a subnet, as Docker does. It skips:

  • routes with the gateway flag (RTF_GATEWAY, 0x2);
  • routes with no interface (*): blackhole, unreachable and prohibit;
  • routes to 0.0.0.0: the default route, as before, and also a gateway-less 0.0.0.0/1 such as dev wg0. Docker skips an unspecified destination the same way.

On-link routes still count. Kiri running inside another engine's container still avoids that container's own eth0 subnet (the comment in Networks::load).

The cost, same trade-off as Docker: a LAN reached only through a gateway, e.g. 172.20.0.0/16 via 192.168.1.254, is no longer avoided automatically. Docker's way out is default-address-pools. Kiri has no pool setting, so the way out is --subnet. This changes which subnets Kiri picks, so it needs your call.

Not in this change:

  • Docker also reserves the nameservers in /etc/resolv.conf.
  • Docker doesn't check IPv6 routes at all, while host_routes6 still does.

Happy to do either here or separately.

How you tested it

Linux, GitHub's ubuntu-24.04 runner. main and this branch are built as static musl binaries, the same way the Speed check builds them. Each engine runs in its own privileged container, and the routes are added in that container's network namespace.

Docker Engine 28.0.4 on the same runner, with OpenVPN-style routes:

0.0.0.0/1 via 10.1.0.1 dev eth0
128.0.0.0/1 via 10.1.0.1 dev eth0
$ docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true --ipv6 --subnet fc00:f853:ccd:e793::/64 kind
fc00:f853:ccd:e793::/64 172.18.0.0/16

Kiri main, same routes, kind's exact command:

base try 1: kind network create -> Error response from daemon: no free subnet left (21 ms)
base try 2: kind network create -> Error response from daemon: no free subnet left (20 ms)
base try 3: kind network create -> Error response from daemon: no free subnet left (19 ms)
plain create: Error response from daemon: no free subnet left
blackhole 172.16/12 + unreachable 10/8: Error response from daemon: no free subnet left

This branch:

head try 1: kind network create -> 0525ccb2… (21 ms)   172.19.0.0/16 fc00:f853:ccd:e793::/64
head try 2: kind network create -> 141329e1… (20 ms)   172.19.0.0/16 fc00:f853:ccd:e793::/64
head try 3: kind network create -> f20eab5c… (19 ms)   172.19.0.0/16 fc00:f853:ccd:e793::/64
container on it, out through the /1 routes: 172.19.0.2/16 outbound-ok
on-link 172.20.0.0/16 added; probe2: 172.21.0.0/16          # on-link routes still count
blackhole 172.16/12 + unreachable 10/8: 172.22.0.0/16
  • only_on_link_routes_reserve_subnets, a new unit test, parses a /proc/net/route with:

    • the OpenVPN def1 pair;
    • a gateway-less WireGuard 0.0.0.0/1;
    • unreachable 10.0.0.0/8 and blackhole 172.16.0.0/12;
    • two on-link LANs and a kiri1 bridge.

    It checks that only the two LANs come back. The test fails if any one of the three new checks is removed (each was reverted separately).

  • cargo test -p kiri --bin kiri network:: passes on Linux (musl) and macOS.

  • cargo clippy -p kiri adds no warnings in the changed lines.

  • rustfmt adds no differences beyond what main already has at the repo's line width.

  • tests/e2e.sh wasn't run for this change. It only touches subnet choice, and the case above runs the API end to end.

Tested on: Linux (Ubuntu 24.04, kernel from GitHub's runner), macOS 27 (unit tests)

How you found it

  • I use Kiri and ran into this myself
  • I found it by reading the code or comparing with Docker
  • Someone else reported it (link the issue above)

AI tools used to write this change: yes. An AI coding assistant helped write and test it. I reviewed and tested every change.

Checklist

  • I have at most 3 pull requests open on Kiri, this one included
  • cargo fmt, cargo clippy and the tests pass
  • It works on macOS, Windows and Linux, or the change only touches one of them (only a Linux host reads its routes; the Mac and Windows VMs return none)

AndreKurait added a commit to AndreKurait/asagiri that referenced this pull request Nov 16, 2026
Port the public Linux routing-table fix while retaining Asagiri's custom
address-pool allocator and host-interface exclusions. Ignore gateway,
blackhole and unspecified-destination routes when reserving connected
subnets, so a full-tunnel route does not cover every pool candidate.

The new regression failed against the original parser. All 20 network
tests now pass, including OpenVPN/WireGuard route-table inputs and default
and custom pools that still avoid the host LAN. Native VPN operation is
not established by these portable tests.

Upstream-PR: libracoder/kiri#351
Upstream-Commit: 4b4df15aad6880965f93a8a2bf8fe21b92678b8c
Co-authored-by: Andre Kurait <akurait@amazon.com>
Signed-off-by: Andre Kurait <akurait@amazon.com>
…ibracoder#228)

Kiri kept every subnet a host route covers out of the address pool. A
route through a gateway, like OpenVPN's full-tunnel 0.0.0.0/1 and
128.0.0.0/1, covers every candidate, so `docker network create` with no
--subnet failed with "no free subnet left", and so did kind.

Since 23.0 Docker reserves only on-link routes (moby#42598). Kiri now does
the same with /proc/net/route: it skips routes through a gateway,
blackhole and unreachable routes (interface "*") and routes to 0.0.0.0.
@sundarshahi
sundarshahi force-pushed the fix/228-pool-ignores-gateway-routes branch from 4b4df15 to 2d5557b Compare November 16, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant