Skip to content

Engine: an --init container's PID 1 keeps none of the agent's descriptors (#248) - #363

Open
sundarshahi wants to merge 1 commit into
libracoder:mainfrom
sundarshahi:fix/248-init-holds-dns-sockets
Open

sundarshahi wants to merge 1 commit into
libracoder:mainfrom
sundarshahi:fix/248-init-holds-dns-sockets

Conversation

@sundarshahi

Copy link
Copy Markdown

What this changes and why

On a Linux host, removing a network while an --init container is running (Compose init: true) leaves the network's DNS listener on <gateway>:53. The next network that gets the same subnet then fails every container start with dns: dns bind: Address in use (os error 98). Docker frees the address when the network is removed.

The cause is the --init container's PID 1. It's a clone of Kiri's agent that never execs, so it keeps a copy of every descriptor the agent had open when it started. That includes each network's DNS socket. The fix closes them in PID 1 once it has forked the program.

Refs #248. This fixes the --init cause of #248, which I reproduced exactly. @dels07 @severinlindenmann, did either of you have an --init or init: true container running when it happened? The reports don't say. The other paths I checked don't leak:

  • a plain run, then removing the network;
  • a failed start (kill_child).

The same descriptor copy also caused a second bug, also reproduced below: with --init, echo x | docker run -i --init alpine cat never ends. PID 1 keeps the write end of its own container's stdin open, so cat never sees end of input. Other containers' stdin pipes are held the same way.

Problem

  1. spawn starts a container with clone3 without CLONE_FILES (crates/kiri-init/src/linux/container.rs:313), so the child gets a copy of every descriptor the agent has open.
  2. Without --init, the program execs, and close-on-exec drops all of them. The agent opens every descriptor close-on-exec: dns.rs:115, sys.rs:65, the signalfd, relays and transport.
  3. With --init, finish_child calls pid1::run (container.rs:1313). That code stays as PID 1 and only its child execs, so PID 1 keeps everything for the container's life:
    • each network's DNS socket. docker network rm → RemoveBridge → Dns::forget_bridge (dns.rs:214) closes only the agent's own copy.
    • the stdin pipe of this container and of others;
    • port-forward connections;
    • the link to the daemon.

Fix

  • In pid1::run, right after fork, PID 1 calls close_range(3, ~0U, 0). After that it only waits for signals with sigtimedwait and reaps children. It needs nothing above stdio:
    • the terminal is already on 0–2;
    • the agent learns of exits through SIGCHLD and waitpid, not through a descriptor PID 1 holds.
  • close_range is the call copy.rs:177 already uses for the same purpose. On a kernel without it (5.7–5.8, since CLONE_INTO_CGROUP needs 5.7), the call fails with ENOSYS and behaves as main does today.
  • Rejected: closing in the agent before clone3. The agent's descriptors have to stay open there, and no clone flag stops descriptors being inherited.

How you tested it

Linux, GitHub's ubuntu-24.04 runner. main and this branch are built as static musl binaries, as the Speed check builds them. Each engine runs in its own privileged container.

The issue's case, 3 times on each. Create n1, start a container on it, start keeper with --init, remove that container and n1, create n2 (same subnet), start a container on n2:

main run 1: n1=172.19.0.0/16 n2=172.19.0.0/16 :53 after rm=[172.19.0.1:53] -> start failed
main run 2: n1=172.19.0.0/16 n2=172.19.0.0/16 :53 after rm=[172.19.0.1:53] -> start failed
main run 3: n1=172.19.0.0/16 n2=172.19.0.0/16 :53 after rm=[172.19.0.1:53] -> start failed
(the error, from an earlier run of the same steps on main:
 docker: Error response from daemon: run 5a1dc494…: dns: dns bind: Address in use (os error 98))
fix  run 1: n1=172.19.0.0/16 n2=172.19.0.0/16 :53 after rm=[]               -> started
fix  run 2: n1=172.19.0.0/16 n2=172.19.0.0/16 :53 after rm=[]               -> started
fix  run 3: n1=172.19.0.0/16 n2=172.19.0.0/16 :53 after rm=[]               -> started

On main, /proc showed the keeper's kiri-init PID 1 as the extra holder. Once keeper was removed, n2 worked again.

Stdin. cat reads from a pipe that closes at 4 s, while an --init container started at 2 s is running:

main: exit=124 after 602s   (cat never saw the end of its input)
fix:  exit=0   after 4s

An --init container's own stdin, 3 times on each:

main run 1-3: echo x | docker run -i --init alpine cat -> printed x, then hung; killed at 20 s (exit 124)
fix  run 1:   echo x | docker run -i --init alpine cat -> exit=0 output=[x] 29 ms
fix  run 2:   echo x | docker run -i --init alpine cat -> exit=0 output=[x] 55 ms
fix  run 3:   echo x | docker run -i --init alpine cat -> exit=0 output=[x] 28 ms

--init still behaves the same:

                 main                          fix
exit code        7                             7
PID 1's fds      0 1 2 3 4 … 24 (25 open)      0 1 2
docker stop      exit 143 in 15 ms             exit 143 in 17 ms
-t               /dev/pts/0 ok                 /dev/pts/0 ok

tests/e2e.sh:

  • New cases:
    • network-rm-init-same-subnet: guards the test itself; the second network has the first one's subnet.
    • network-rm-init-dns: the issue's case.
    • init-stdin-eof: echo x | docker run -i --init alpine cat.
  • init-stdin-eof runs cat under the container's own timeout 10, so a regression fails with 143 instead of hanging the suite.
  • The full suite ran against both engines on the runner, as root, from this branch's tests/e2e.sh. Both runs completed.
    • Failing on main only: network-rm-init-dns and init-stdin-eof.
    • Failing on this branch only: none.
    • Both have the same other failures, which come from the runner rather than Kiri: the Mac-only machine, Finder and *.kiri.localhost checks, and host.docker.internal inside a nested engine.
  • cargo clippy -p kiri-init (Linux, musl) has the same warnings on main and on this branch.
  • rustfmt on pid1.rs shows no new differences.

Tested on: Linux (Ubuntu 24.04, GitHub runner's kernel). The Mac and Windows run the same kiri-init in their VMs. I didn't run them by hand.

How you found it

  • I use Kiri and ran into this myself
  • I found it by reading the code or comparing with Docker
  • Someone else reported it (link the issue above)

AI tools used to write this change: yes. An AI coding assistant helped write and test it. I reviewed and tested every change.

Checklist

  • I have at most 3 pull requests open on Kiri, this one included
  • cargo fmt, cargo clippy and the tests pass
  • It works on macOS, Windows and Linux, or the change only touches one of them

…tors (libracoder#248)

With --init, a container's PID 1 is Kiri's own code. It is a clone of the
agent and never execs, so close-on-exec never applied to it. It kept a
copy of every descriptor the agent had open when it started, for as long
as the container ran:

- each network's DNS socket. After `docker network rm`, that copy still
  held the gateway's port 53, so the next network on the same subnet
  failed every start with "dns bind: Address in use".
- its own container's stdin pipe, so `echo x | docker run -i --init
  alpine cat` never saw the end of its input.
- other containers' pipes, the port forwards' connections and the
  link to the daemon.

PID 1 now closes everything above stdio once it has forked the program.
It needs none of them. The program never had them: they close when it
execs.
@sundarshahi
sundarshahi force-pushed the fix/248-init-holds-dns-sockets branch from 1ad66cc to 3a7ecfa Compare November 16, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant