diff --git a/basefiles/sysctl.d/99-dstack.conf b/basefiles/sysctl.d/99-dstack.conf new file mode 100644 index 00000000..fc8db7b1 --- /dev/null +++ b/basefiles/sysctl.d/99-dstack.conf @@ -0,0 +1,6 @@ +# SPDX-FileCopyrightText: 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +# Increase conntrack table for high-concurrency gateway/proxy workloads. +# Default 262144 is insufficient when proxying >100K concurrent connections. +net.netfilter.nf_conntrack_max = 2097152 diff --git a/docs/deployment.md b/docs/deployment.md index 463e1aeb..bb9a1f22 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -524,6 +524,36 @@ The CID range conflicts with existing VMs. cid_pool_size = 1000 ``` +### High-concurrency deployments: conntrack table full + +When running Gateway with many concurrent connections (>100K), the host's conntrack table may fill up, causing silent packet drops: + +``` +dmesg: nf_conntrack: table full, dropping packet +``` + +Each proxied connection creates multiple conntrack entries (client→gateway, gateway→WireGuard→backend). The default `nf_conntrack_max` (typically 262,144) is insufficient for high-concurrency gateways. + +**Fix:** + +```bash +# Check current limit +sysctl net.netfilter.nf_conntrack_max + +# Increase for production (persistent) +echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/99-dstack.conf +echo "net.netfilter.nf_conntrack_buckets = 262144" >> /etc/sysctl.d/99-dstack.conf +sysctl -p /etc/sysctl.d/99-dstack.conf +``` + +Also increase inside bridge-mode CVMs if they handle many connections: + +```bash +sysctl -w net.netfilter.nf_conntrack_max=524288 +``` + +**Sizing rule of thumb:** Set `nf_conntrack_max` to at least 4× your target concurrent connection count (each connection may use 2-3 conntrack entries across NAT/bridge layers). + ### Error: Operation not permitted when building guest image Ubuntu 23.10+ restricts unprivileged user namespaces: