From 0d845320e1d946dce250649cdfb8cf26ab1d0e60 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 2 Feb 2026 06:33:25 +0000 Subject: [PATCH 1/2] feat: add conntrack sysctl config for CVM guest (2M max) --- basefiles/sysctl.d/99-dstack.conf | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 basefiles/sysctl.d/99-dstack.conf diff --git a/basefiles/sysctl.d/99-dstack.conf b/basefiles/sysctl.d/99-dstack.conf new file mode 100644 index 00000000..fc8db7b1 --- /dev/null +++ b/basefiles/sysctl.d/99-dstack.conf @@ -0,0 +1,6 @@ +# SPDX-FileCopyrightText: 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 + +# Increase conntrack table for high-concurrency gateway/proxy workloads. +# Default 262144 is insufficient when proxying >100K concurrent connections. +net.netfilter.nf_conntrack_max = 2097152 From f5dc822800f21e3df949549abc3d8d5fc77de44a Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Mon, 2 Feb 2026 06:33:28 +0000 Subject: [PATCH 2/2] docs: add conntrack tuning guide for high-concurrency gateway deployments --- docs/deployment.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/deployment.md b/docs/deployment.md index 463e1aeb..bb9a1f22 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -524,6 +524,36 @@ The CID range conflicts with existing VMs. cid_pool_size = 1000 ``` +### High-concurrency deployments: conntrack table full + +When running Gateway with many concurrent connections (>100K), the host's conntrack table may fill up, causing silent packet drops: + +``` +dmesg: nf_conntrack: table full, dropping packet +``` + +Each proxied connection creates multiple conntrack entries (client→gateway, gateway→WireGuard→backend). The default `nf_conntrack_max` (typically 262,144) is insufficient for high-concurrency gateways. + +**Fix:** + +```bash +# Check current limit +sysctl net.netfilter.nf_conntrack_max + +# Increase for production (persistent) +echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/99-dstack.conf +echo "net.netfilter.nf_conntrack_buckets = 262144" >> /etc/sysctl.d/99-dstack.conf +sysctl -p /etc/sysctl.d/99-dstack.conf +``` + +Also increase inside bridge-mode CVMs if they handle many connections: + +```bash +sysctl -w net.netfilter.nf_conntrack_max=524288 +``` + +**Sizing rule of thumb:** Set `nf_conntrack_max` to at least 4× your target concurrent connection count (each connection may use 2-3 conntrack entries across NAT/bridge layers). + ### Error: Operation not permitted when building guest image Ubuntu 23.10+ restricts unprivileged user namespaces: