Skip to content

Commit 94de01a

Browse files
Danatharclaude
andauthored
fix: stop "too many open files" warnings on Kubernetes (kubeadm) servers (#663)
* fix: raise inotify limits on kubeadm nodes Pods' root processes share one per-user inotify budget, and the kernel default of 128 instances runs out on a node running workloads. Once it does, a new systemd user manager (started by sudo, for one) fails with "Too many open files" and broadcasts it to every console. Raise the instance and watch limits in the kubeadm sysctl drop-in that containerd.service already applies. Closes #658 Hive-Run: #658 Hive-Plan: approved-direction on #658 Hive-Spec: server#658#approved-direction Assisted-by: Claude Opus 5.5 via Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com> * fix: set inotify watch limit to the kernel ceiling so it never lowers it Since Linux 5.11 the default max_user_watches scales with RAM (capped at 1048576), so 524288 lowered the limit on nodes with ~60 GB or more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 1d820b6 commit 94de01a

3 files changed

Lines changed: 11 additions & 2 deletions

File tree

‎docs/skills/kubeadm-sysext.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,9 @@ but never newer. Patch releases change neither constraint.
8787
`imports` glob on a config that pre-dates drop-in glob support, see
8888
[Migration](#migration)),
8989
`modprobe overlay br_netfilter`, then `systemd-sysctl 90-kubeadm.conf`
90-
(`ip_forward`, `bridge-nf-call-ip{,6}tables`). kubeadm needs containerd
91-
running, so these hold before any preflight.
90+
(`ip_forward`, `bridge-nf-call-ip{,6}tables`, and raised `fs.inotify`
91+
instance and watch limits, since pods share root's per-user budget).
92+
kubeadm needs containerd running, so these hold before any preflight.
9293
- containerd root is `/var/lib/containerd` (`RequiresMountsFor=` orders it after
9394
a per-node mount), `SystemdCgroup = true`, sandbox `registry.k8s.io/pause:3.10.1`,
9495
registry `config_path = /etc/containerd/certs.d`.

‎files/kubeadm/sysext/sysctl-90-kubeadm.conf‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,9 @@
22
net.ipv4.ip_forward = 1
33
net.bridge.bridge-nf-call-iptables = 1
44
net.bridge.bridge-nf-call-ip6tables = 1
5+
# Every pod's root processes share one per-user inotify budget; the kernel
6+
# default of 128 instances runs out on a busy node and breaks anything new that
7+
# needs one (systemd user managers log "Too many open files", e.g. under sudo).
8+
fs.inotify.max_user_instances = 8192
9+
# 1048576 is the kernel's own RAM-scaled ceiling (5.11+), so this only ever raises it.
10+
fs.inotify.max_user_watches = 1048576

‎tests/unit/test_kubeadm_sysext.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,8 @@ def test_host_modules_and_sysctls() -> None:
168168
sysctl = (SRC / "sysctl-90-kubeadm.conf").read_text()
169169
for key in ("net.ipv4.ip_forward", "net.bridge.bridge-nf-call-iptables", "net.bridge.bridge-nf-call-ip6tables"):
170170
assert f"{key} = 1" in sysctl
171+
assert "fs.inotify.max_user_instances = 8192" in sysctl
172+
assert "fs.inotify.max_user_watches = 1048576" in sysctl
171173

172174

173175
def test_kubelet_units_follow_kubeadm() -> None:

0 commit comments

Comments
 (0)