Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions basefiles/sysctl.d/99-dstack.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# SPDX-FileCopyrightText: 2025 Phala Network <dstack@phala.network>
# SPDX-License-Identifier: Apache-2.0

# Increase conntrack table for high-concurrency gateway/proxy workloads.
# Default 262144 is insufficient when proxying >100K concurrent connections.
net.netfilter.nf_conntrack_max = 2097152
30 changes: 30 additions & 0 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,36 @@ The CID range conflicts with existing VMs.
cid_pool_size = 1000
```

### High-concurrency deployments: conntrack table full

When running Gateway with many concurrent connections (>100K), the host's conntrack table may fill up, causing silent packet drops:

```
dmesg: nf_conntrack: table full, dropping packet
```

Each proxied connection creates multiple conntrack entries (client→gateway, gateway→WireGuard→backend). The default `nf_conntrack_max` (typically 262,144) is insufficient for high-concurrency gateways.

**Fix:**

```bash
# Check current limit
sysctl net.netfilter.nf_conntrack_max

# Increase for production (persistent)
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/99-dstack.conf
echo "net.netfilter.nf_conntrack_buckets = 262144" >> /etc/sysctl.d/99-dstack.conf
sysctl -p /etc/sysctl.d/99-dstack.conf
```

Also increase inside bridge-mode CVMs if they handle many connections:

```bash
sysctl -w net.netfilter.nf_conntrack_max=524288
```

**Sizing rule of thumb:** Set `nf_conntrack_max` to at least 4× your target concurrent connection count (each connection may use 2-3 conntrack entries across NAT/bridge layers).

### Error: Operation not permitted when building guest image

Ubuntu 23.10+ restricts unprivileged user namespaces:
Expand Down