chinmaybaikar · Sep 15, 2026 · Sep 14, 2026 · Sep 15, 2026
@@ -175,7 +175,7 @@ A hardened, redundant route-based IPsec (IKEv2) VPN terminating on one or two Ub
 
 [StrongSwan Site-to-Site VPN for Crusoe Cloud](./strongswan-ipsec/)
 
-An Ansible-managed, encrypted IPsec site-to-site VPN between a Crusoe Cloud region and a remote site — another Crusoe region, or Azure/GCP/AWS — with VMs on both sides communicating via their real (non-NAT'd) IP addresses over a GRE-over-FOU overlay or plain routes. You fill in an inventory and five values, and one command preflights connectivity, installs a VAES-capable kernel where it pays, and configures gateways and clients with defaults tuned for a managed cloud peer. Measured at 2.4 Gbps with one gateway per side rising to 20.7 Gbps with five, and 8.5 Gbps through a single VM holding two tunnels; also configures managed Kubernetes nodes via a DaemonSet.
+An Ansible-managed, encrypted IPsec site-to-site VPN between a Crusoe Cloud region and a remote site — another Crusoe region, or Azure/GCP/AWS — with VMs on both sides communicating via their real (non-NAT'd) IP addresses over a GRE-over-FOU overlay. You fill in an inventory and five values, and one command preflights connectivity, installs a VAES-capable kernel where it pays, and configures gateways and clients with defaults tuned for a managed cloud peer. Measured at 2.4 Gbps with one gateway per side rising to 20.7 Gbps with five, and 8.5 Gbps through a single VM holding two tunnels; also configures managed Kubernetes nodes via a DaemonSet.
 
 ## Contributing
 

@@ -335,9 +335,6 @@ Stated plainly, so nobody builds on sand.
 - **BGP and FRR were never brought up against a real peer.** Since 10 Gbps
   through Azure requires BGP (§3.1 item 5), that path needs validating on
   first use.
-- **The `route` transport was never validated end to end.** Crusoe port
-  security was on throughout, and a foreign-source packet was confirmed
-  dropped by the fabric. `gre_fou` is the tested path and the default.
 - The Crusoe-to-Crusoe curve in §1 **is measured**, on the shipped playbook,
   1 through 5 gateways per side. An earlier *derived* estimate in this
   document claimed ~8 Gbps per VM and 2 VMs for 10 Gbps; the measurement

@@ -8,8 +8,8 @@ standalone VMs with Ansible and managed Kubernetes nodes with a DaemonSet.
 
 - Subnet-to-subnet routing over encrypted IPsec, plus full-tunnel mode
 - Managed K8s support — the DaemonSet configures nodes, no SSH needed
-- **Selectable client transport** — GRE-over-FOU overlay (default) or plain
-  static routes
+- **GRE-over-FOU overlay** to local Crusoe clients, or a gateway with no local
+  clients that just forwards LAN-to-LAN
 - **Named crypto profiles**, including `gcmaes256`
 - **Multiple tunnels per gateway** with ECMP, for peers that publish more than
   one outer address — most managed cloud VPNs do in their redundant mode
@@ -215,12 +215,6 @@ Scaled out, with several gateway VMs:
 > tell a dead gateway from a live one. It fails safe — leaving routes alone
 > rather than removing them — but you lose automatic failover.
 
-**For `vpn_client_transport: route` only, and no playbook can do it:** Crusoe
-must **disable port security** on every gateway VM's vNIC, or add
-allowed-address-pairs covering `vpn_remote_subnet`. A gateway in `route` mode
-forwards decrypted packets whose source is a *remote* private IP, and the SDN
-drops those by default. The symptom is a healthy tunnel with no return traffic.
-
 ## Quick Start
 
 ### 1. Configure — two files
@@ -310,27 +304,17 @@ To measure throughput, use [bandwidth-test](../bandwidth-test/).
 | Value | Meaning | Prerequisite |
 |---|---|---|
 | `gre_fou` | GRE-over-FOU overlay (**default**) | none |
-| `route` | plain static routes via the gateway's private IP | **port security disabled on each gateway vNIC** |
 | `none` | gateway carries no local clients; forwards its own LAN subnet | none |
 
-`vpn_use_gre: true/false` still works as a deprecated alias.
+The client role only ever brings up GRE-over-FOU, so it refuses to configure
+against a gateway it resolves as `none` rather than silently bringing up a
+GRE device with no working peer.
 
-`gre_fou` stays the default because `route` needs a manual network change no
-playbook can make. Where `route` is allowed it is better:
+`vpn_use_gre: true/false` still works as a deprecated alias.
 
-| | `gre_fou` | `route` |
-|---|---|---|
-| Client devices | N GRE devices + FOU sockets | none |
-| Client MTU | 1400 | 1400 |
-| Gateway setup | FOU module, multipoint GRE, one neighbour entry per host in the CIDR, 2 policy tables | plain FIB forwarding |
-| **GRO on the uplink** | **must be OFF** — see troubleshooting | unaffected |
-| **Client CIDR size** | a `/20` is 4094 neighbour entries; a `/16` is refused | any size |
-| Flow entropy on the fabric | one FOU 4-tuple per client↔gateway pair unless `vpn_fou_sport_auto` | the real client 5-tuples |
-| Dead-gateway detection | needs the ICMP probe | `fib_multipath_use_neigh` handles VM death for free |
-
-On `gre_fou`, set `vpn_fou_sport_auto: true` for better flow spread without the
-port-security change — the kernel then hashes the outer FOU source port per
-inner flow.
+Set `vpn_fou_sport_auto: true` for better flow spread across the fabric — the
+kernel then hashes the outer FOU source port per inner flow instead of pinning
+every client↔gateway pair to one 4-tuple.
 
 ## Crypto profiles
 
@@ -595,7 +579,7 @@ Gateway (`roles/vpn_gateway/defaults/main.yml` documents every one):
 | Variable | Default | Purpose |
 |----------|---------|---------|
 | `vpn_psk` | — | IKE pre-shared key |
-| `vpn_client_transport` | `""` → `gre_fou`/`none` | `gre_fou`, `route`, or `none` |
+| `vpn_client_transport` | `""` → `gre_fou` | `gre_fou` or `none` |
 | `vpn_client_cidr` | — | subnet of VMs/nodes behind this gateway |
 | `vpn_local_subnet` / `vpn_remote_subnet` | — | IPsec traffic selectors |
 | `vpn_remote_gw_ip` | — | peer public IP; the fallback when `vpn_remote_addrs` is empty |
@@ -638,10 +622,7 @@ DaemonSet (`k8s/vpn-client.yaml`): `GATEWAY_IPS`, `TRANSPORT`, `REMOTE_CIDRS`,
   per host: a `/20` is 4094, a `/16` is refused by the role.
 - **`gre_fou` requires GRO off on every uplink.** Host-wide setting, affecting
   all traffic on that NIC.
-- **`route` transport needs port security disabled**, and is **the one path not
-  validated live** — it renders and passes static checks, but port security was
-  enabled throughout testing. Treat the first `route` deployment as supervised.
-- **BGP is not validated live** either; it is not needed for a
+- **BGP is not validated live**; it is not needed for a
   Crusoe-to-Crusoe pairing. All Azure figures quoted come from Microsoft's
   published tables, not measurement.
 - **SNAT and multi-gateway ECMP are mutually exclusive.** The role refuses the
@@ -700,7 +681,6 @@ kubectl logs -n kube-system ds/vpn-client
 | **Ping and UDP fine, TCP collapses to a few Mbit/s** | **GRO on the physical NIC.** It coalesces inbound FOU packets that then cannot be re-encapsulated, so they are dropped — measured **3.65 Mbps vs 4590 Mbps**. `ethtool -K <uplink> gro off` on gateways **and** clients; the role does this automatically for `gre_fou`. Look for `UdpInErrors` climbing on the receiving gateway. |
 | TCP stalls only for full-size packets | MSS clamped **above** the path MTU. `iptables --set-mss` raises as well as lowers. Leave `vpn_mss_clamp_value` empty so it is derived. |
 | Tunnel up, no traffic | Routes through xfrm? Mark rules present? (`ip rule show`) |
-| `route` mode: tunnel up, no return traffic | **Port security still enabled on the gateway vNIC.** The usual cause. |
 | N tunnels but no more throughput | `fib_multipath_hash_policy` = 1? Does `vpn_remote_addrs` have more than one address? Are per-SA byte counters even? |
 | Throughput per flow stuck near 1 Gbps | Client TCP buffers. `vpn_client_tcp_tuning` and `tcp_rmem` max ≥ 64 MB. |
 | **Tunnel healthy, ping clean, TCP retransmitting hard** | **Anti-replay is discarding reordered packets.** `grep XfrmInStateSeqError /proc/net/xfrm_stat` — if it climbs, check `cat /sys/class/net/<uplink>/queues/rx-0/rps_flow_cnt`. RFS hands one SA's packets between CPUs so they arrive out of order. Measured 18,624 retransmits with RFS on versus 106 with it off. Keep `vpn_disable_rfs: true`. |

@@ -92,15 +92,10 @@ vpn_client_cidr: "<your-crusoe-subnet-cidr>"
 # vpn_bgp_accept: ["<peer-subnet-cidr>"]
 # (vpn_bgp_local_asn and vpn_bgp_peer_addrs are per gateway - see host_vars)
 
-# --- Dropping GRE ---------------------------------------------------------
-# "route" removes the GRE overlay: no GRO constraint, and no per-host
-# neighbour entry, so client CIDRs larger than a /20 become possible.
-#
-# PREREQUISITE NO PLAYBOOK CAN MEET: Crusoe NetEng must DISABLE PORT SECURITY
-# on every gateway VM's vNIC. Without it the decrypted return traffic is
-# dropped by the fabric and the tunnel looks healthy while nothing arrives.
-# That manual step is why gre_fou is the default.
-# vpn_client_transport: "route"
+# --- No local clients ------------------------------------------------------
+# A gateway with no Crusoe VMs behind it - just LAN-to-LAN forwarding between
+# the two sites, no GRE overlay at all.
+# vpn_client_transport: "none"
 
 # --- Kernel upgrade -------------------------------------------------------
 # ON by default. It installs linux-image-generic-6.11 and reboots, but only

@@ -39,16 +39,8 @@ vpn_gateway_hosts: "{{ groups['vpn_gateways'] | default([]) }}"
 # the customer states the remote CIDR exactly once.
 
 # ---------------------------------------------------------------------------
-# Transport - must match the gateway
-# ---------------------------------------------------------------------------
-#   gre_fou  GRE-over-FOU overlay (default)
-#   route    plain static routes via the gateway's private IP. Requires
-#            Crusoe port security to be DISABLED on the GATEWAY's vNIC.
-# "" = inherit from the first gateway's own configuration, which is the safe
-# default because a mismatch silently breaks the datapath.
-vpn_client_transport: ""
-
 # GRE-over-FOU - must match the gateway
+# ---------------------------------------------------------------------------
 # Must match the gateway. See the vpn_gateway role for the MTU maths: the
 # binding constraint is vpn_xfrm_mtu (1400), not the 1500-byte uplink.
 vpn_gre_mtu: 1400
@@ -57,10 +49,6 @@ vpn_fou_port: 9473
 vpn_fou_sport_auto: false
 vpn_fou_port_count: 1
 
-# route transport: MTU applied to the ECMP route so TCP shrinks without
-# relying on PMTUD. Azure's documented tunnel MTU is 1400.
-vpn_client_route_mtu: 1400
-
 # ---------------------------------------------------------------------------
 # Health - clients do not run BGP, so they probe instead
 # ---------------------------------------------------------------------------
@@ -79,10 +67,6 @@ vpn_client_probe_timeout: 1
 # sit idle. The single most missable line on the client side.
 vpn_multipath_hash_policy: 1
 
-# route transport: skip a nexthop whose neighbour entry is unreachable. Free
-# dead-gateway detection, no agent involved.
-vpn_multipath_use_neigh: 1
-
 # With ECMP the reply may arrive from a different gateway than the request
 # left through, which strict reverse-path filtering rejects.
 vpn_rp_filter: 2

@@ -36,21 +36,15 @@
     vpn_local_ip: "{{ ansible_default_ipv4.address }}"
   tags: [setup, verify]
 
-# Transport must match the gateway, and a mismatch breaks the datapath
-# silently, so it is inherited from the gateway unless overridden here.
+# GRE-over-FOU is the only client transport this role supports, but the
+# gateway may be running "none" (no local clients) instead - inherited here
+# rather than assumed, so that mismatch is caught below instead of silently
+# producing a dead GRE device that reports UP regardless of the peer.
 - name: Resolve the transport
   vars:
     _gw: "{{ vpn_gateway_host_list[0] }}"
-    _from_gateway: >-
-      {{ hostvars[_gw].vpn_transport
-         | default(hostvars[_gw].vpn_client_transport | default('', true), true) }}
   ansible.builtin.set_fact:
-    vpn_transport: >-
-      {{ vpn_client_transport if vpn_client_transport
-         else (_from_gateway if _from_gateway
-               else ('gre_fou'
-                     if (hostvars[_gw].vpn_use_gre | default(true) | bool)
-                     else 'route')) }}
+    vpn_transport: "{{ hostvars[_gw].vpn_transport | default('gre_fou', true) }}"
   tags: [setup, teardown, verify]
 
 - name: Resolve the remote CIDRs
@@ -69,19 +63,21 @@
 - name: Validate configuration
   ansible.builtin.assert:
     that:
-      - vpn_transport in ['gre_fou', 'route']
+      - vpn_transport == 'gre_fou'
       - vpn_client_remote_cidrs | length > 0
       - vpn_client_paths | rejectattr('gateway_ip') | list | length == 0
     fail_msg: >-
-      Could not resolve every gateway's private IP. The client role reads it
-      from each gateway's vpn_local_gw_ip, which the vpn_gateway role sets as
-      a fact - so running the gateways play first (as site.yml does) is
-      enough. If you want to configure clients on their own, for example with
-      "--limit vpn_clients", set vpn_local_gw_ip explicitly on each gateway
-      host in the inventory or host_vars:
-      "gw-1  ansible_host=<public-ip>  vpn_local_gw_ip=<private-ip>".
-      Also check vpn_client_transport is gre_fou or route and that
-      vpn_remote_subnets is set.
+      Either the gateway's transport is not gre_fou ({{ vpn_gateway_host_list[0] }}
+      resolved to "{{ vpn_transport }}" - this role has nothing to bring up
+      against a gateway running vpn_client_transport=none, since that gateway
+      serves no local clients), or a gateway's private IP could not be
+      resolved. The client role reads that from each gateway's
+      vpn_local_gw_ip, which the vpn_gateway role sets as a fact - so running
+      the gateways play first (as site.yml does) is enough. If you want to
+      configure clients on their own, for example with "--limit vpn_clients",
+      set vpn_local_gw_ip explicitly on each gateway host in the inventory or
+      host_vars: "gw-1  ansible_host=<public-ip>  vpn_local_gw_ip=<private-ip>".
+      Also check vpn_remote_subnets is set.
   tags: [setup, verify]
 
 # ---------------------------------------------------------------------------
@@ -153,7 +149,6 @@
   community.general.modprobe:
     name: fou
     state: present
-  when: vpn_transport == 'gre_fou'
   tags: [setup]
 
 - name: Install NIC tuning script
@@ -238,7 +233,6 @@
   loop_control:
     label: "{{ item.dev }}"
   changed_when: false
-  when: vpn_transport == 'gre_fou'
   tags: [setup, verify]
 
 - name: Count nexthops on the first remote route
@@ -261,7 +255,7 @@
   register: vpn_client_gro_state
   changed_when: false
   failed_when: false
-  when: vpn_transport == 'gre_fou' and (vpn_fou_disable_gro | bool)
+  when: vpn_fou_disable_gro | bool
   tags: [setup, verify]
 
 - name: "WARNING: GRO is still on and will destroy TCP"
@@ -271,7 +265,6 @@
       coalesced and dropped; UDP and ICMP keep working while TCP collapses
       (measured 3.65 Mbps vs 4590 Mbps). See the README troubleshooting table.
   when:
-    - vpn_transport == 'gre_fou'
     - vpn_fou_disable_gro | bool
     - (vpn_client_gro_state.stdout | default('') | trim) == 'on'
   tags: [setup, verify]
@@ -298,8 +291,7 @@
   ansible.builtin.debug:
     msg: >-
       Client {{ inventory_hostname }} ({{ vpn_local_ip }}):
-      transport={{ vpn_transport }}
-      | {{ vpn_client_paths | length }} path(s) to
+      {{ vpn_client_paths | length }} path(s) to
       {{ vpn_client_paths | map(attribute='gateway_ip') | unique | join(', ') }}
       | ECMP nexthops installed={{ vpn_client_nexthops.stdout | trim }}
       | remote={{ vpn_client_remote_cidrs | join(', ') }}

@@ -5,13 +5,6 @@
 # the same pair of hosts takes ONE gateway and the others sit idle. The setup
 # looks correct and performs like a single gateway.
 net.ipv4.fib_multipath_hash_policy = {{ vpn_multipath_hash_policy }}
-{% if vpn_transport == 'route' %}
-
-# Skip a nexthop whose neighbour entry has gone unreachable. Free dead-gateway
-# detection for route transport (a GRE device stays UP regardless, which is why
-# gre_fou needs the probe instead).
-net.ipv4.fib_multipath_use_neigh = {{ vpn_multipath_use_neigh }}
-{% endif %}
 
 # The reply may arrive from a different gateway than the request left through.
 net.ipv4.conf.all.rp_filter = {{ vpn_rp_filter }}

@@ -22,11 +22,7 @@ alive_count=0
 
 {% for p in vpn_client_paths %}
 if ping -c 1 -W "$TIMEOUT" -n -q {{ p.gateway_ip }} >/dev/null 2>&1; then
-{% if vpn_transport == 'route' %}
-    alive+=(nexthop via {{ p.gateway_ip }} weight 1)
-{% else %}
     alive+=(nexthop dev {{ p.dev }} weight 1)
-{% endif %}
     alive_count=$(( alive_count + 1 ))
 fi
 {% endfor %}
@@ -44,7 +40,7 @@ previous=$(cat "$STATE_FILE" 2>/dev/null || true)
 
 for cidr in $REMOTE_CIDRS; do
     # shellcheck disable=SC2086
-    ip route replace "$cidr" "${alive[@]}" {{ 'mtu ' ~ vpn_client_route_mtu if (vpn_transport == 'route' and vpn_client_route_mtu) else '' }}
+    ip route replace "$cidr" "${alive[@]}"
 done
 printf '%s' "$desired" > "$STATE_FILE"
 echo "vpn-client-health: $alive_count/{{ vpn_client_paths | length }} gateway(s) up; routes rebuilt"
@@ -8,26 +8,16 @@
 # Usage: vpn-client.sh {up|down}
 set -u
 
-{#- ECMP nexthop spec: device list for gre_fou, via-IP list for route. #}
-{%- if vpn_transport == 'route' -%}
-{%- set NH = vpn_client_paths | map(attribute='gateway_ip')
-      | map('regex_replace', '^(.*)$', 'nexthop via \\1 weight 1') | join(' ') -%}
-{%- if vpn_client_paths | length == 1 -%}
-{%- set NH = 'via ' ~ vpn_client_paths[0].gateway_ip -%}
-{%- endif -%}
-{%- set MTU_ARG = 'mtu ' ~ vpn_client_route_mtu if vpn_client_route_mtu else '' -%}
-{%- else -%}
+{#- ECMP nexthop spec: one device per gateway path. #}
 {%- set NH = vpn_client_paths | map(attribute='dev')
       | map('regex_replace', '^(.*)$', 'nexthop dev \\1 weight 1') | join(' ') -%}
 {%- if vpn_client_paths | length == 1 -%}
 {%- set NH = 'dev ' ~ vpn_client_paths[0].dev -%}
-{%- endif -%}
-{%- set MTU_ARG = '' -%}
 {%- endif %}
 
-# transport = {{ vpn_transport }}, paths = {{ vpn_client_paths | length }}
+# paths = {{ vpn_client_paths | length }}
 {% for p in vpn_client_paths %}
-#   path {{ p.index }}: {{ p.gateway_host }} ({{ p.gateway_ip }}){% if vpn_transport == 'gre_fou' %} via {{ p.dev }} on UDP {{ p.fou_port }}{% endif %}
+#   path {{ p.index }}: {{ p.gateway_host }} ({{ p.gateway_ip }}) via {{ p.dev }} on UDP {{ p.fou_port }}
 
 {% endfor %}
 REMOTE_CIDRS="{{ vpn_client_remote_cidrs | join(' ') }}"
@@ -86,7 +76,7 @@ up_tunnels() {
 up_routes() {
     for cidr in $REMOTE_CIDRS; do
         # shellcheck disable=SC2086
-        ip route replace "$cidr" {{ NH }} {{ MTU_ARG }}
+        ip route replace "$cidr" {{ NH }}
     done
 }
 
@@ -100,7 +90,7 @@ do_up() {
     up_tunnels
 {% endif %}
     up_routes
-    echo "vpn-client up: {{ vpn_client_paths | length }} path(s), transport {{ vpn_transport }}"
+    echo "vpn-client up: {{ vpn_client_paths | length }} path(s)"
 }
 
 do_down() {