diff --git a/README.md b/README.md index 9cc8a66..5c530dd 100644 --- a/README.md +++ b/README.md @@ -175,7 +175,7 @@ A hardened, redundant route-based IPsec (IKEv2) VPN terminating on one or two Ub [StrongSwan Site-to-Site VPN for Crusoe Cloud](./strongswan-ipsec/) -An Ansible-managed, encrypted IPsec site-to-site VPN between a Crusoe Cloud region and a remote site — another Crusoe region, or Azure/GCP/AWS — with VMs on both sides communicating via their real (non-NAT'd) IP addresses over a GRE-over-FOU overlay or plain routes. You fill in an inventory and five values, and one command preflights connectivity, installs a VAES-capable kernel where it pays, and configures gateways and clients with defaults tuned for a managed cloud peer. Measured at 2.4 Gbps with one gateway per side rising to 20.7 Gbps with five, and 8.5 Gbps through a single VM holding two tunnels; also configures managed Kubernetes nodes via a DaemonSet. +An Ansible-managed, encrypted IPsec site-to-site VPN between a Crusoe Cloud region and a remote site — another Crusoe region, or Azure/GCP/AWS — with VMs on both sides communicating via their real (non-NAT'd) IP addresses over a GRE-over-FOU overlay. You fill in an inventory and five values, and one command preflights connectivity, installs a VAES-capable kernel where it pays, and configures gateways and clients with defaults tuned for a managed cloud peer. Measured at 2.4 Gbps with one gateway per side rising to 20.7 Gbps with five, and 8.5 Gbps through a single VM holding two tunnels; also configures managed Kubernetes nodes via a DaemonSet. ## Contributing diff --git a/strongswan-ipsec/AZURE-10G-GUIDE.md b/strongswan-ipsec/AZURE-10G-GUIDE.md index 8850480..64874dd 100644 --- a/strongswan-ipsec/AZURE-10G-GUIDE.md +++ b/strongswan-ipsec/AZURE-10G-GUIDE.md @@ -335,9 +335,6 @@ Stated plainly, so nobody builds on sand. - **BGP and FRR were never brought up against a real peer.** Since 10 Gbps through Azure requires BGP (§3.1 item 5), that path needs validating on first use. -- **The `route` transport was never validated end to end.** Crusoe port - security was on throughout, and a foreign-source packet was confirmed - dropped by the fabric. `gre_fou` is the tested path and the default. - The Crusoe-to-Crusoe curve in §1 **is measured**, on the shipped playbook, 1 through 5 gateways per side. An earlier *derived* estimate in this document claimed ~8 Gbps per VM and 2 VMs for 10 Gbps; the measurement diff --git a/strongswan-ipsec/README.md b/strongswan-ipsec/README.md index f0c17a8..d36a679 100644 --- a/strongswan-ipsec/README.md +++ b/strongswan-ipsec/README.md @@ -8,8 +8,8 @@ standalone VMs with Ansible and managed Kubernetes nodes with a DaemonSet. - Subnet-to-subnet routing over encrypted IPsec, plus full-tunnel mode - Managed K8s support — the DaemonSet configures nodes, no SSH needed -- **Selectable client transport** — GRE-over-FOU overlay (default) or plain - static routes +- **GRE-over-FOU overlay** to local Crusoe clients, or a gateway with no local + clients that just forwards LAN-to-LAN - **Named crypto profiles**, including `gcmaes256` - **Multiple tunnels per gateway** with ECMP, for peers that publish more than one outer address — most managed cloud VPNs do in their redundant mode @@ -215,12 +215,6 @@ Scaled out, with several gateway VMs: > tell a dead gateway from a live one. It fails safe — leaving routes alone > rather than removing them — but you lose automatic failover. -**For `vpn_client_transport: route` only, and no playbook can do it:** Crusoe -must **disable port security** on every gateway VM's vNIC, or add -allowed-address-pairs covering `vpn_remote_subnet`. A gateway in `route` mode -forwards decrypted packets whose source is a *remote* private IP, and the SDN -drops those by default. The symptom is a healthy tunnel with no return traffic. - ## Quick Start ### 1. Configure — two files @@ -310,27 +304,17 @@ To measure throughput, use [bandwidth-test](../bandwidth-test/). | Value | Meaning | Prerequisite | |---|---|---| | `gre_fou` | GRE-over-FOU overlay (**default**) | none | -| `route` | plain static routes via the gateway's private IP | **port security disabled on each gateway vNIC** | | `none` | gateway carries no local clients; forwards its own LAN subnet | none | -`vpn_use_gre: true/false` still works as a deprecated alias. +The client role only ever brings up GRE-over-FOU, so it refuses to configure +against a gateway it resolves as `none` rather than silently bringing up a +GRE device with no working peer. -`gre_fou` stays the default because `route` needs a manual network change no -playbook can make. Where `route` is allowed it is better: +`vpn_use_gre: true/false` still works as a deprecated alias. -| | `gre_fou` | `route` | -|---|---|---| -| Client devices | N GRE devices + FOU sockets | none | -| Client MTU | 1400 | 1400 | -| Gateway setup | FOU module, multipoint GRE, one neighbour entry per host in the CIDR, 2 policy tables | plain FIB forwarding | -| **GRO on the uplink** | **must be OFF** — see troubleshooting | unaffected | -| **Client CIDR size** | a `/20` is 4094 neighbour entries; a `/16` is refused | any size | -| Flow entropy on the fabric | one FOU 4-tuple per client↔gateway pair unless `vpn_fou_sport_auto` | the real client 5-tuples | -| Dead-gateway detection | needs the ICMP probe | `fib_multipath_use_neigh` handles VM death for free | - -On `gre_fou`, set `vpn_fou_sport_auto: true` for better flow spread without the -port-security change — the kernel then hashes the outer FOU source port per -inner flow. +Set `vpn_fou_sport_auto: true` for better flow spread across the fabric — the +kernel then hashes the outer FOU source port per inner flow instead of pinning +every client↔gateway pair to one 4-tuple. ## Crypto profiles @@ -595,7 +579,7 @@ Gateway (`roles/vpn_gateway/defaults/main.yml` documents every one): | Variable | Default | Purpose | |----------|---------|---------| | `vpn_psk` | — | IKE pre-shared key | -| `vpn_client_transport` | `""` → `gre_fou`/`none` | `gre_fou`, `route`, or `none` | +| `vpn_client_transport` | `""` → `gre_fou` | `gre_fou` or `none` | | `vpn_client_cidr` | — | subnet of VMs/nodes behind this gateway | | `vpn_local_subnet` / `vpn_remote_subnet` | — | IPsec traffic selectors | | `vpn_remote_gw_ip` | — | peer public IP; the fallback when `vpn_remote_addrs` is empty | @@ -638,10 +622,7 @@ DaemonSet (`k8s/vpn-client.yaml`): `GATEWAY_IPS`, `TRANSPORT`, `REMOTE_CIDRS`, per host: a `/20` is 4094, a `/16` is refused by the role. - **`gre_fou` requires GRO off on every uplink.** Host-wide setting, affecting all traffic on that NIC. -- **`route` transport needs port security disabled**, and is **the one path not - validated live** — it renders and passes static checks, but port security was - enabled throughout testing. Treat the first `route` deployment as supervised. -- **BGP is not validated live** either; it is not needed for a +- **BGP is not validated live**; it is not needed for a Crusoe-to-Crusoe pairing. All Azure figures quoted come from Microsoft's published tables, not measurement. - **SNAT and multi-gateway ECMP are mutually exclusive.** The role refuses the @@ -700,7 +681,6 @@ kubectl logs -n kube-system ds/vpn-client | **Ping and UDP fine, TCP collapses to a few Mbit/s** | **GRO on the physical NIC.** It coalesces inbound FOU packets that then cannot be re-encapsulated, so they are dropped — measured **3.65 Mbps vs 4590 Mbps**. `ethtool -K gro off` on gateways **and** clients; the role does this automatically for `gre_fou`. Look for `UdpInErrors` climbing on the receiving gateway. | | TCP stalls only for full-size packets | MSS clamped **above** the path MTU. `iptables --set-mss` raises as well as lowers. Leave `vpn_mss_clamp_value` empty so it is derived. | | Tunnel up, no traffic | Routes through xfrm? Mark rules present? (`ip rule show`) | -| `route` mode: tunnel up, no return traffic | **Port security still enabled on the gateway vNIC.** The usual cause. | | N tunnels but no more throughput | `fib_multipath_hash_policy` = 1? Does `vpn_remote_addrs` have more than one address? Are per-SA byte counters even? | | Throughput per flow stuck near 1 Gbps | Client TCP buffers. `vpn_client_tcp_tuning` and `tcp_rmem` max ≥ 64 MB. | | **Tunnel healthy, ping clean, TCP retransmitting hard** | **Anti-replay is discarding reordered packets.** `grep XfrmInStateSeqError /proc/net/xfrm_stat` — if it climbs, check `cat /sys/class/net//queues/rx-0/rps_flow_cnt`. RFS hands one SA's packets between CPUs so they arrive out of order. Measured 18,624 retransmits with RFS on versus 106 with it off. Keep `vpn_disable_rfs: true`. | diff --git a/strongswan-ipsec/ansible/group_vars/all.yml b/strongswan-ipsec/ansible/group_vars/all.yml index 6193fbd..ad18385 100644 --- a/strongswan-ipsec/ansible/group_vars/all.yml +++ b/strongswan-ipsec/ansible/group_vars/all.yml @@ -92,15 +92,10 @@ vpn_client_cidr: "" # vpn_bgp_accept: [""] # (vpn_bgp_local_asn and vpn_bgp_peer_addrs are per gateway - see host_vars) -# --- Dropping GRE --------------------------------------------------------- -# "route" removes the GRE overlay: no GRO constraint, and no per-host -# neighbour entry, so client CIDRs larger than a /20 become possible. -# -# PREREQUISITE NO PLAYBOOK CAN MEET: Crusoe NetEng must DISABLE PORT SECURITY -# on every gateway VM's vNIC. Without it the decrypted return traffic is -# dropped by the fabric and the tunnel looks healthy while nothing arrives. -# That manual step is why gre_fou is the default. -# vpn_client_transport: "route" +# --- No local clients ------------------------------------------------------ +# A gateway with no Crusoe VMs behind it - just LAN-to-LAN forwarding between +# the two sites, no GRE overlay at all. +# vpn_client_transport: "none" # --- Kernel upgrade ------------------------------------------------------- # ON by default. It installs linux-image-generic-6.11 and reboots, but only diff --git a/strongswan-ipsec/ansible/roles/vpn_client/defaults/main.yml b/strongswan-ipsec/ansible/roles/vpn_client/defaults/main.yml index 035541f..a93e451 100644 --- a/strongswan-ipsec/ansible/roles/vpn_client/defaults/main.yml +++ b/strongswan-ipsec/ansible/roles/vpn_client/defaults/main.yml @@ -39,16 +39,8 @@ vpn_gateway_hosts: "{{ groups['vpn_gateways'] | default([]) }}" # the customer states the remote CIDR exactly once. # --------------------------------------------------------------------------- -# Transport - must match the gateway -# --------------------------------------------------------------------------- -# gre_fou GRE-over-FOU overlay (default) -# route plain static routes via the gateway's private IP. Requires -# Crusoe port security to be DISABLED on the GATEWAY's vNIC. -# "" = inherit from the first gateway's own configuration, which is the safe -# default because a mismatch silently breaks the datapath. -vpn_client_transport: "" - # GRE-over-FOU - must match the gateway +# --------------------------------------------------------------------------- # Must match the gateway. See the vpn_gateway role for the MTU maths: the # binding constraint is vpn_xfrm_mtu (1400), not the 1500-byte uplink. vpn_gre_mtu: 1400 @@ -57,10 +49,6 @@ vpn_fou_port: 9473 vpn_fou_sport_auto: false vpn_fou_port_count: 1 -# route transport: MTU applied to the ECMP route so TCP shrinks without -# relying on PMTUD. Azure's documented tunnel MTU is 1400. -vpn_client_route_mtu: 1400 - # --------------------------------------------------------------------------- # Health - clients do not run BGP, so they probe instead # --------------------------------------------------------------------------- @@ -79,10 +67,6 @@ vpn_client_probe_timeout: 1 # sit idle. The single most missable line on the client side. vpn_multipath_hash_policy: 1 -# route transport: skip a nexthop whose neighbour entry is unreachable. Free -# dead-gateway detection, no agent involved. -vpn_multipath_use_neigh: 1 - # With ECMP the reply may arrive from a different gateway than the request # left through, which strict reverse-path filtering rejects. vpn_rp_filter: 2 diff --git a/strongswan-ipsec/ansible/roles/vpn_client/tasks/main.yml b/strongswan-ipsec/ansible/roles/vpn_client/tasks/main.yml index 44dd01e..d312945 100644 --- a/strongswan-ipsec/ansible/roles/vpn_client/tasks/main.yml +++ b/strongswan-ipsec/ansible/roles/vpn_client/tasks/main.yml @@ -36,21 +36,15 @@ vpn_local_ip: "{{ ansible_default_ipv4.address }}" tags: [setup, verify] -# Transport must match the gateway, and a mismatch breaks the datapath -# silently, so it is inherited from the gateway unless overridden here. +# GRE-over-FOU is the only client transport this role supports, but the +# gateway may be running "none" (no local clients) instead - inherited here +# rather than assumed, so that mismatch is caught below instead of silently +# producing a dead GRE device that reports UP regardless of the peer. - name: Resolve the transport vars: _gw: "{{ vpn_gateway_host_list[0] }}" - _from_gateway: >- - {{ hostvars[_gw].vpn_transport - | default(hostvars[_gw].vpn_client_transport | default('', true), true) }} ansible.builtin.set_fact: - vpn_transport: >- - {{ vpn_client_transport if vpn_client_transport - else (_from_gateway if _from_gateway - else ('gre_fou' - if (hostvars[_gw].vpn_use_gre | default(true) | bool) - else 'route')) }} + vpn_transport: "{{ hostvars[_gw].vpn_transport | default('gre_fou', true) }}" tags: [setup, teardown, verify] - name: Resolve the remote CIDRs @@ -69,19 +63,21 @@ - name: Validate configuration ansible.builtin.assert: that: - - vpn_transport in ['gre_fou', 'route'] + - vpn_transport == 'gre_fou' - vpn_client_remote_cidrs | length > 0 - vpn_client_paths | rejectattr('gateway_ip') | list | length == 0 fail_msg: >- - Could not resolve every gateway's private IP. The client role reads it - from each gateway's vpn_local_gw_ip, which the vpn_gateway role sets as - a fact - so running the gateways play first (as site.yml does) is - enough. If you want to configure clients on their own, for example with - "--limit vpn_clients", set vpn_local_gw_ip explicitly on each gateway - host in the inventory or host_vars: - "gw-1 ansible_host= vpn_local_gw_ip=". - Also check vpn_client_transport is gre_fou or route and that - vpn_remote_subnets is set. + Either the gateway's transport is not gre_fou ({{ vpn_gateway_host_list[0] }} + resolved to "{{ vpn_transport }}" - this role has nothing to bring up + against a gateway running vpn_client_transport=none, since that gateway + serves no local clients), or a gateway's private IP could not be + resolved. The client role reads that from each gateway's + vpn_local_gw_ip, which the vpn_gateway role sets as a fact - so running + the gateways play first (as site.yml does) is enough. If you want to + configure clients on their own, for example with "--limit vpn_clients", + set vpn_local_gw_ip explicitly on each gateway host in the inventory or + host_vars: "gw-1 ansible_host= vpn_local_gw_ip=". + Also check vpn_remote_subnets is set. tags: [setup, verify] # --------------------------------------------------------------------------- @@ -153,7 +149,6 @@ community.general.modprobe: name: fou state: present - when: vpn_transport == 'gre_fou' tags: [setup] - name: Install NIC tuning script @@ -238,7 +233,6 @@ loop_control: label: "{{ item.dev }}" changed_when: false - when: vpn_transport == 'gre_fou' tags: [setup, verify] - name: Count nexthops on the first remote route @@ -261,7 +255,7 @@ register: vpn_client_gro_state changed_when: false failed_when: false - when: vpn_transport == 'gre_fou' and (vpn_fou_disable_gro | bool) + when: vpn_fou_disable_gro | bool tags: [setup, verify] - name: "WARNING: GRO is still on and will destroy TCP" @@ -271,7 +265,6 @@ coalesced and dropped; UDP and ICMP keep working while TCP collapses (measured 3.65 Mbps vs 4590 Mbps). See the README troubleshooting table. when: - - vpn_transport == 'gre_fou' - vpn_fou_disable_gro | bool - (vpn_client_gro_state.stdout | default('') | trim) == 'on' tags: [setup, verify] @@ -298,8 +291,7 @@ ansible.builtin.debug: msg: >- Client {{ inventory_hostname }} ({{ vpn_local_ip }}): - transport={{ vpn_transport }} - | {{ vpn_client_paths | length }} path(s) to + {{ vpn_client_paths | length }} path(s) to {{ vpn_client_paths | map(attribute='gateway_ip') | unique | join(', ') }} | ECMP nexthops installed={{ vpn_client_nexthops.stdout | trim }} | remote={{ vpn_client_remote_cidrs | join(', ') }} diff --git a/strongswan-ipsec/ansible/roles/vpn_client/templates/sysctl-vpn-client.conf.j2 b/strongswan-ipsec/ansible/roles/vpn_client/templates/sysctl-vpn-client.conf.j2 index 0bdfab8..173dd80 100644 --- a/strongswan-ipsec/ansible/roles/vpn_client/templates/sysctl-vpn-client.conf.j2 +++ b/strongswan-ipsec/ansible/roles/vpn_client/templates/sysctl-vpn-client.conf.j2 @@ -5,13 +5,6 @@ # the same pair of hosts takes ONE gateway and the others sit idle. The setup # looks correct and performs like a single gateway. net.ipv4.fib_multipath_hash_policy = {{ vpn_multipath_hash_policy }} -{% if vpn_transport == 'route' %} - -# Skip a nexthop whose neighbour entry has gone unreachable. Free dead-gateway -# detection for route transport (a GRE device stays UP regardless, which is why -# gre_fou needs the probe instead). -net.ipv4.fib_multipath_use_neigh = {{ vpn_multipath_use_neigh }} -{% endif %} # The reply may arrive from a different gateway than the request left through. net.ipv4.conf.all.rp_filter = {{ vpn_rp_filter }} diff --git a/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client-health.sh.j2 b/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client-health.sh.j2 index da0b310..677ebc4 100644 --- a/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client-health.sh.j2 +++ b/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client-health.sh.j2 @@ -22,11 +22,7 @@ alive_count=0 {% for p in vpn_client_paths %} if ping -c 1 -W "$TIMEOUT" -n -q {{ p.gateway_ip }} >/dev/null 2>&1; then -{% if vpn_transport == 'route' %} - alive+=(nexthop via {{ p.gateway_ip }} weight 1) -{% else %} alive+=(nexthop dev {{ p.dev }} weight 1) -{% endif %} alive_count=$(( alive_count + 1 )) fi {% endfor %} @@ -44,7 +40,7 @@ previous=$(cat "$STATE_FILE" 2>/dev/null || true) for cidr in $REMOTE_CIDRS; do # shellcheck disable=SC2086 - ip route replace "$cidr" "${alive[@]}" {{ 'mtu ' ~ vpn_client_route_mtu if (vpn_transport == 'route' and vpn_client_route_mtu) else '' }} + ip route replace "$cidr" "${alive[@]}" done printf '%s' "$desired" > "$STATE_FILE" echo "vpn-client-health: $alive_count/{{ vpn_client_paths | length }} gateway(s) up; routes rebuilt" diff --git a/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client.sh.j2 b/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client.sh.j2 index 090a340..11c3fbf 100644 --- a/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client.sh.j2 +++ b/strongswan-ipsec/ansible/roles/vpn_client/templates/vpn-client.sh.j2 @@ -8,26 +8,16 @@ # Usage: vpn-client.sh {up|down} set -u -{#- ECMP nexthop spec: device list for gre_fou, via-IP list for route. #} -{%- if vpn_transport == 'route' -%} -{%- set NH = vpn_client_paths | map(attribute='gateway_ip') - | map('regex_replace', '^(.*)$', 'nexthop via \\1 weight 1') | join(' ') -%} -{%- if vpn_client_paths | length == 1 -%} -{%- set NH = 'via ' ~ vpn_client_paths[0].gateway_ip -%} -{%- endif -%} -{%- set MTU_ARG = 'mtu ' ~ vpn_client_route_mtu if vpn_client_route_mtu else '' -%} -{%- else -%} +{#- ECMP nexthop spec: one device per gateway path. #} {%- set NH = vpn_client_paths | map(attribute='dev') | map('regex_replace', '^(.*)$', 'nexthop dev \\1 weight 1') | join(' ') -%} {%- if vpn_client_paths | length == 1 -%} {%- set NH = 'dev ' ~ vpn_client_paths[0].dev -%} -{%- endif -%} -{%- set MTU_ARG = '' -%} {%- endif %} -# transport = {{ vpn_transport }}, paths = {{ vpn_client_paths | length }} +# paths = {{ vpn_client_paths | length }} {% for p in vpn_client_paths %} -# path {{ p.index }}: {{ p.gateway_host }} ({{ p.gateway_ip }}){% if vpn_transport == 'gre_fou' %} via {{ p.dev }} on UDP {{ p.fou_port }}{% endif %} +# path {{ p.index }}: {{ p.gateway_host }} ({{ p.gateway_ip }}) via {{ p.dev }} on UDP {{ p.fou_port }} {% endfor %} REMOTE_CIDRS="{{ vpn_client_remote_cidrs | join(' ') }}" @@ -86,7 +76,7 @@ up_tunnels() { up_routes() { for cidr in $REMOTE_CIDRS; do # shellcheck disable=SC2086 - ip route replace "$cidr" {{ NH }} {{ MTU_ARG }} + ip route replace "$cidr" {{ NH }} done } @@ -100,7 +90,7 @@ do_up() { up_tunnels {% endif %} up_routes - echo "vpn-client up: {{ vpn_client_paths | length }} path(s), transport {{ vpn_transport }}" + echo "vpn-client up: {{ vpn_client_paths | length }} path(s)" } do_down() { diff --git a/strongswan-ipsec/ansible/roles/vpn_gateway/defaults/main.yml b/strongswan-ipsec/ansible/roles/vpn_gateway/defaults/main.yml index 1ef6687..59d4987 100644 --- a/strongswan-ipsec/ansible/roles/vpn_gateway/defaults/main.yml +++ b/strongswan-ipsec/ansible/roles/vpn_gateway/defaults/main.yml @@ -150,8 +150,8 @@ vpn_force_encap: true # understates throughput and drops real user traffic. # # Auto now resolves to: -# start whenever this gateway serves clients (gre_fou or route), or has -# more than one tunnel - i.e. every case where laziness costs traffic +# start whenever this gateway serves clients (gre_fou), or has more than +# one tunnel - i.e. every case where laziness costs traffic # trap only for a plain point-to-point gateway with no local clients vpn_start_action: "" @@ -207,15 +207,9 @@ vpn_replay_window: 1024 # Client transport (how local Crusoe VMs / K8s nodes reach this gateway) # --------------------------------------------------------------------------- # gre_fou GRE-over-FOU overlay (== the deprecated vpn_use_gre: true) -# route plain static routes, no encapsulation. REQUIRES Crusoe port -# security to be DISABLED on this gateway's vNIC (or allowed- -# address-pairs covering vpn_remote_subnet). Without that, -# decrypted return traffic is dropped by the SDN. See the README. # none this gateway carries no local clients (== vpn_use_gre: false) # -# "" = gre_fou. That is the default because "route" REQUIRES Crusoe NetEng to -# disable port security on every gateway vNIC by hand, and no playbook can do -# that for you. +# "" = gre_fou. # # Left as "" rather than the literal "gre_fou" so the deprecated vpn_use_gre # spelling stays reachable: hardcoding it here would make "vpn_use_gre: false" @@ -258,15 +252,6 @@ vpn_fou_sport_auto: true # Device name stays exactly "gre-vpn" while this is 1. vpn_fou_port_count: 1 -# How "route" transport gets decrypted return traffic past the SDN: -# port_security_disabled NetEng disabled port security / added allowed- -# address-pairs. Stateless, scales. Recommended. -# snat gateway SNATs remote->client traffic to its own -# address. No network-team dependency, but needs -# conntrack per packet and is INCOMPATIBLE with -# multi-gateway ECMP. Not for >10 Gbps. -vpn_route_return: "port_security_disabled" - # SNAT decrypted traffic heading for the internet. Required for full-tunnel # internet egress, but SNAT is stateful, so it CANNOT be combined with # multi-gateway ECMP. Off by default: the shipped datapath is stateless so @@ -333,7 +318,7 @@ vpn_rp_filter: 2 # # Derived values (see vpn_mss_effective in tasks/main.yml): # gre_fou vpn_gre_mtu - 40 (1400 - 40 = 1360) -# route/none vpn_xfrm_mtu - 40 (1400 - 40 = 1360) +# none vpn_xfrm_mtu - 40 (1400 - 40 = 1360) # # Both come to 1360 because vpn_gre_mtu is 1400: the gateway strips the GRE # wrapper before encrypting, so the tunnel MTU, not the GRE MTU, is the @@ -360,7 +345,7 @@ vpn_stateful_forward_rule: false # ens3 GRO off -> 4590 Mbps (~1250x) # # This is correctness, not tuning, so it is applied by vpn-network.sh whenever -# the transport is gre_fou - not gated behind vpn_perf_tuning. The route +# the transport is gre_fou - not gated behind vpn_perf_tuning. The none # transport has no FOU and keeps GRO on. vpn_fou_disable_gro: true diff --git a/strongswan-ipsec/ansible/roles/vpn_gateway/tasks/main.yml b/strongswan-ipsec/ansible/roles/vpn_gateway/tasks/main.yml index e6fcf11..16a1cc5 100644 --- a/strongswan-ipsec/ansible/roles/vpn_gateway/tasks/main.yml +++ b/strongswan-ipsec/ansible/roles/vpn_gateway/tasks/main.yml @@ -3,11 +3,11 @@ # vpn_gateway role # # Terminates N IPsec tunnels to a remote site and delivers traffic to local -# Crusoe clients. Three client transports, selected by vpn_client_transport: +# Crusoe clients. Two client transports, selected by vpn_client_transport: # # gre_fou GRE-over-FOU overlay (default; == the old vpn_use_gre: true) -# route plain static routes - needs Crusoe port security DISABLED # none no local clients; forward this gateway's own LAN subnet +# (== the old vpn_use_gre: false) # # With every new variable at its default this behaves exactly as the # single-tunnel version always did. See README.md, and AZURE-10G-GUIDE.md for @@ -38,7 +38,7 @@ - name: Derive transport booleans and traffic selectors ansible.builtin.set_fact: - vpn_serves_clients: "{{ vpn_transport in ['gre_fou', 'route'] }}" + vpn_serves_clients: "{{ vpn_transport == 'gre_fou' }}" vpn_child_local_ts: >- {{ '0.0.0.0/0' if (vpn_traffic_selectors_any | bool) else vpn_local_subnet }} vpn_child_remote_ts: >- @@ -132,14 +132,13 @@ that: - (vpn_tunnel_count_effective | int) >= 1 - (vpn_tunnel_count_effective | int) <= 32 - - vpn_transport in ['gre_fou', 'route', 'none'] + - vpn_transport in ['gre_fou', 'none'] - not (vpn_serves_clients | bool) or (vpn_client_cidr is defined) - - vpn_route_return in ['port_security_disabled', 'snat'] - vpn_crypto_profile in vpn_crypto_profiles fail_msg: >- Invalid vpn_gateway configuration. tunnel_count must be 1-32, - vpn_client_transport must be gre_fou/route/none, vpn_client_cidr is - required for gre_fou and route, and vpn_crypto_profile must be one of + vpn_client_transport must be gre_fou/none, vpn_client_cidr is + required for gre_fou, and vpn_crypto_profile must be one of {{ vpn_crypto_profiles.keys() | list }}. tags: [setup, verify] @@ -166,12 +165,11 @@ that: - not (vpn_disable_conntrack | bool) or not (vpn_stateful_forward_rule | bool) - not (vpn_disable_conntrack | bool) or not (vpn_snat_internet_egress | bool) - - not (vpn_route_return == 'snat') or not (vpn_disable_conntrack | bool) fail_msg: >- Conntrack interlock violated. NOTRACK'd packets can never match a ctstate rule and can never be SNAT'd, so vpn_disable_conntrack=true - requires vpn_stateful_forward_rule=false, vpn_snat_internet_egress=false - and vpn_route_return != snat. Note also that SNAT is stateful, so it + requires vpn_stateful_forward_rule=false and + vpn_snat_internet_egress=false. Note also that SNAT is stateful, so it cannot be combined with multi-gateway ECMP at all - see the "Known limitations" section of README.md. tags: [setup, verify] @@ -208,17 +206,6 @@ - (vpn_remote_addrs | length) <= 1 tags: [setup, verify] -- name: "WARNING: route transport needs port security disabled" - ansible.builtin.debug: - msg: >- - vpn_client_transport=route forwards decrypted packets whose source is a - REMOTE private IP out of this gateway's vNIC. Crusoe port security drops - those. Confirm NetEng has disabled port security on this VM's vNIC (or - added allowed-address-pairs covering {{ vpn_remote_subnet }}) before - expecting return traffic to arrive. - when: vpn_transport == 'route' and vpn_route_return == 'port_security_disabled' - tags: [setup, verify] - # Multipoint GRE has no signalling, so the gateway needs one PERMANENT # neighbour entry per host in vpn_client_cidr. That does not scale: a /20 is # 4094 entries (fine, ~seconds), a /16 is 65534 (exceeds the neighbour table @@ -239,8 +226,7 @@ net.ipv4.neigh.default.gc_thresh3 ({{ vpn_neigh_gc_thresh3 }}). The table would overflow and an arbitrary subset of clients would silently never work. Either narrow vpn_client_cidr to the range that actually holds - clients, raise vpn_neigh_gc_thresh1/2/3, or switch to - vpn_client_transport=route, which needs no neighbour table at all. + clients, or raise vpn_neigh_gc_thresh1/2/3. when: vpn_transport == 'gre_fou' tags: [setup, verify] @@ -249,8 +235,7 @@ msg: >- vpn_client_cidr {{ vpn_client_cidr }} means {{ vpn_neigh_needed }} permanent neighbour entries on every gateway. That works, but it adds - deploy time and memory on each gateway. The route transport needs none - of it. + deploy time and memory on each gateway. when: - vpn_transport == 'gre_fou' - (vpn_neigh_needed | int) > 8000 @@ -509,8 +494,8 @@ when: vpn_transport == 'gre_fou' tags: [setup] -# Covers route transport in both directions, and internet egress for -# full-tunnel mode, without needing per-tunnel rules. +# Covers internet egress for full-tunnel mode, without needing per-tunnel +# rules. - name: "Allow forwarding: LAN <-> IPsec" ansible.builtin.iptables: chain: FORWARD @@ -543,11 +528,11 @@ tags: [setup] # The internet-egress SNAT below matches "source = remote subnet, out = uplink". -# In route transport, traffic heading BACK to local clients also leaves via the -# uplink with a remote source address, so it would match and be NAT'd - which -# both hides the real remote IPs and makes the path stateful, breaking -# multi-gateway ECMP. This RETURN exempts client-bound traffic, and must be -# added BEFORE the SNAT rule. +# Traffic heading BACK to local GRE clients also leaves via the uplink with a +# remote source address, so it would match and be NAT'd - which both hides the +# real remote IPs and makes the path stateful, breaking multi-gateway ECMP. +# This RETURN exempts client-bound traffic, and must be added BEFORE the +# SNAT rule. - name: "Exempt client-bound traffic from the internet SNAT" ansible.builtin.iptables: table: nat @@ -580,19 +565,6 @@ when: vpn_snat_internet_egress | bool tags: [setup] -- name: "SNAT remote traffic towards clients (route transport fallback)" - ansible.builtin.iptables: - table: nat - chain: POSTROUTING - out_interface: "{{ vpn_primary_iface }}" - source: "{{ vpn_remote_subnet }}" - destination: "{{ vpn_client_cidr }}" - jump: SNAT - to_source: "{{ vpn_local_gw_ip }}" - comment: "VPN: SNAT return traffic past port security" - when: vpn_transport == 'route' and vpn_route_return == 'snat' - tags: [setup] - - name: Allow established/related forwarding ansible.builtin.iptables: chain: FORWARD diff --git a/strongswan-ipsec/ansible/roles/vpn_gateway/templates/vpn-network.sh.j2 b/strongswan-ipsec/ansible/roles/vpn_gateway/templates/vpn-network.sh.j2 index c163e35..7126e0d 100644 --- a/strongswan-ipsec/ansible/roles/vpn_gateway/templates/vpn-network.sh.j2 +++ b/strongswan-ipsec/ansible/roles/vpn_gateway/templates/vpn-network.sh.j2 @@ -191,17 +191,10 @@ up_overlay_routing() { # Client traffic may be destined anywhere (full tunnel), so mark it and route # it through a dedicated table. The gateway's own default route is untouched. up_mark_routing() { -{% if vpn_transport == 'gre_fou' %} {% for g in gre_devs %} iptables -t mangle -D PREROUTING -i {{ g.dev }} -j MARK --set-mark {{ vpn_fwmark }} 2>/dev/null || true iptables -t mangle -A PREROUTING -i {{ g.dev }} -j MARK --set-mark {{ vpn_fwmark }} {% endfor %} -{% else %} - # route transport: match client traffic on the uplink, but never traffic - # that stays inside the local subnet (that includes this gateway's own IP). - iptables -t mangle -D PREROUTING -i "$UPLINK" -s {{ vpn_client_cidr }} ! -d {{ vpn_client_cidr }} -j MARK --set-mark {{ vpn_fwmark }} 2>/dev/null || true - iptables -t mangle -A PREROUTING -i "$UPLINK" -s {{ vpn_client_cidr }} ! -d {{ vpn_client_cidr }} -j MARK --set-mark {{ vpn_fwmark }} -{% endif %} ip rule del fwmark {{ vpn_fwmark }} lookup {{ vpn_mark_table }} 2>/dev/null || true ip rule add fwmark {{ vpn_fwmark }} lookup {{ vpn_mark_table }} ip route replace default {{ XFRM_NH }} table {{ vpn_mark_table }} @@ -229,10 +222,6 @@ up_mss_clamp() { # importantly, enforces the statelessness that multi-gateway ECMP depends on: # a reply may legitimately return through a DIFFERENT gateway. up_notrack() { -{% if vpn_transport == 'route' %} - iptables -t raw -D PREROUTING -i "$UPLINK" -s {{ vpn_client_cidr }} -j NOTRACK 2>/dev/null || true - iptables -t raw -A PREROUTING -i "$UPLINK" -s {{ vpn_client_cidr }} -j NOTRACK -{% endif %} {% for g in gre_devs %} iptables -t raw -D PREROUTING -i {{ g.dev }} -j NOTRACK 2>/dev/null || true iptables -t raw -A PREROUTING -i {{ g.dev }} -j NOTRACK @@ -286,9 +275,6 @@ do_down() { done {% endif %} {% if vpn_disable_conntrack | bool %} -{% if vpn_transport == 'route' %} - iptables -t raw -D PREROUTING -i "$UPLINK" -s {{ vpn_client_cidr }} -j NOTRACK 2>/dev/null || true -{% endif %} {% for g in gre_devs %} iptables -t raw -D PREROUTING -i {{ g.dev }} -j NOTRACK 2>/dev/null || true {% endfor %} @@ -297,13 +283,9 @@ do_down() { {% endfor %} {% endif %} {% if vpn_serves_clients | bool %} -{% if vpn_transport == 'gre_fou' %} {% for g in gre_devs %} iptables -t mangle -D PREROUTING -i {{ g.dev }} -j MARK --set-mark {{ vpn_fwmark }} 2>/dev/null || true {% endfor %} -{% else %} - iptables -t mangle -D PREROUTING -i "$UPLINK" -s {{ vpn_client_cidr }} ! -d {{ vpn_client_cidr }} -j MARK --set-mark {{ vpn_fwmark }} 2>/dev/null || true -{% endif %} ip rule del fwmark {{ vpn_fwmark }} lookup {{ vpn_mark_table }} 2>/dev/null || true ip route flush table {{ vpn_mark_table }} 2>/dev/null || true {% endif %} diff --git a/strongswan-ipsec/k8s/vpn-client.yaml b/strongswan-ipsec/k8s/vpn-client.yaml index 8d9646e..df94505 100644 --- a/strongswan-ipsec/k8s/vpn-client.yaml +++ b/strongswan-ipsec/k8s/vpn-client.yaml @@ -3,10 +3,7 @@ # one or more VPN gateways. Same job as the vpn_client Ansible role, for # managed clusters where Ansible cannot reach the nodes. # -# Two transports, matching the Ansible role: -# gre_fou GRE-over-FOU overlay (default) -# route plain static routes via the gateway private IPs. Requires Crusoe -# port security to be DISABLED on every GATEWAY's vNIC. +# Reaches the gateways over a GRE-over-FOU overlay, matching the Ansible role. # # Traffic is ECMP'd across all gateways, hashed per flow. Nothing here is # stateful, so a reply may come back through a different gateway - which is @@ -28,7 +25,6 @@ data: # behaviour. Several entries = ECMP across them. GATEWAY_IPS: "" - # gre_fou | route - must match the gateways TRANSPORT: "gre_fou" # GRE-over-FOU settings - must match the gateways (vpn_gre_key, @@ -44,9 +40,6 @@ data: # Extra FOU ports per gateway, for kernels without "encap-sport auto". FOU_PORT_COUNT: "1" - # route transport: MTU on the ECMP route. Azure's documented tunnel MTU. - ROUTE_MTU: "1400" - # GRO MUST be off on the node's uplink when using gre_fou. Generic Receive # Offload coalesces inbound FOU/UDP packets; the coalesced packet cannot be # re-encapsulated and is dropped. UDP and ICMP are unaffected, so the tunnel @@ -67,8 +60,6 @@ data: # every flow takes ONE gateway and the others sit idle. The cluster looks # healthy and performs like a single gateway. MULTIPATH_HASH_POLICY: "1" - # route transport: skip nexthops whose ARP entry has gone unreachable. - MULTIPATH_USE_NEIGH: "1" RP_FILTER: "2" # Long-fat-pipe TCP. At 50 ms RTT, 10 Gbps needs ~62 MB in flight, but the @@ -166,9 +157,6 @@ spec: sysctl -qw "net.ipv4.fib_multipath_hash_policy=${MULTIPATH_HASH_POLICY}" 2>/dev/null || true sysctl -qw "net.ipv4.conf.all.rp_filter=${RP_FILTER}" 2>/dev/null || true sysctl -qw "net.ipv4.conf.default.rp_filter=${RP_FILTER}" 2>/dev/null || true - if [ "$TRANSPORT" = "route" ]; then - sysctl -qw "net.ipv4.fib_multipath_use_neigh=${MULTIPATH_USE_NEIGH}" 2>/dev/null || true - fi if [ "${TCP_TUNING:-true}" = "true" ]; then modprobe tcp_bbr 2>/dev/null || true sysctl -qw "net.core.rmem_max=${RMEM_MAX}" 2>/dev/null || true @@ -222,15 +210,11 @@ spec: # re-hash: nothing here is stateful, so a flow that moves to # another gateway keeps working. sync_routes() { - local i nh=() up=0 mtu=() + local i nh=() up=0 for (( i=0; i<${#PATH_DEV[@]}; i++ )); do ping -c 1 -W "${PROBE_TIMEOUT:-1}" -n -q "${PATH_GW[$i]}" >/dev/null 2>&1 || continue - if [ "$TRANSPORT" = "route" ]; then - nh+=(nexthop via "${PATH_GW[$i]}" weight 1) - else - ip link show "${PATH_DEV[$i]}" 2>/dev/null | grep -q UP || continue - nh+=(nexthop dev "${PATH_DEV[$i]}" weight 1) - fi + ip link show "${PATH_DEV[$i]}" 2>/dev/null | grep -q UP || continue + nh+=(nexthop dev "${PATH_DEV[$i]}" weight 1) up=$(( up + 1 )) done @@ -238,7 +222,6 @@ spec: echo "WARNING: no gateway answered - leaving routes alone" >&2 return 0 fi - [ "$TRANSPORT" = "route" ] && [ -n "${ROUTE_MTU:-}" ] && mtu=(mtu "$ROUTE_MTU") local desired="${nh[*]}" if [ "$desired" != "${LAST_NH:-}" ]; then @@ -246,7 +229,7 @@ spec: LAST_NH="$desired" fi for cidr in $REMOTE_CIDRS; do - ip route replace "$cidr" "${nh[@]}" "${mtu[@]}" 2>/dev/null || true + ip route replace "$cidr" "${nh[@]}" 2>/dev/null || true done }