XDP data-plane armed-only hardening: race-free RATE (X1) + SYN-cookie TX cap (X3)

CHANGELOG.md

@@ -9,6 +9,7 @@ All notable changes to this project are documented here, following
 - Speedtest now runs providers sequentially, measures each download over the full window, and reports the fastest clean provider — fixing wildly variable, under-reporting results on fast links.
 
 ### Fixed
+- XDP data-plane armed-only hardening (AS214806 M1 follow-up; the two data-plane bugs from the arming re-triage). **X1 — race-free per-source rate limiter:** the `RATE` map was a CPU-shared `LruHashMap` whose token-bucket read-modify-write in `over_rate` was non-atomic, so under a multi-CPU flood (one source spraying spoofed source ports, RSS-spread across CPUs) the per-source pps limit leaked ~N×. A `bpf_spin_lock` was spiked first (the exact fix) but the verifier rejects it on this toolchain (aya-ebpf 0.1.1's `#[map]` emits legacy non-BTF maps, which `bpf_spin_lock` requires), so `RATE` is now an `LruPerCpuHashMap` — each CPU an isolated bucket, RMW race-free without a lock. Trade-off (documented): the effective per-source limit becomes up to ~`N_cpus ×` configured (looser, never tighter); userspace per-CPU summing for accurate per-source reporting is a follow-on. **X3 — SYN-cookie `XDP_TX` mint-rate cap:** the in-kernel SYN-cookie fast path minted a SipHash SYN-ACK per inbound SYN via `XDP_TX` with no ceiling — a gain-1 reflector at the attacker's rate. A global per-CPU token bucket (`TX_BUDGET`) now caps SYN-ACK mints (burst scaled to the configured rate, `min(2×rate, MAX)`); over budget the SYN takes its normal `XDP_PASS`/drop verdict and increments `blackwall_xdp_syn_cookies_txcapped_total`. A `syn-cookie-tx-cap=<pps>` `xdp` directive sets it with a conservative default (1000 pps, never unlimited), and the cap is seeded before the cookie port gate is armed so the fast path can never run unbounded. Both paths are dormant unless XDP mitigation / `cookie-ports` are configured (no M0 impact). Proven by `BPF_PROG_TEST_RUN` + veth lab gates.
 - M1 arming hardening (control plane) — closes the three arming-safety follow-ups from the M1 interlock. (**#194**) the boot-time rehydrate path (RTBH/FlowSpec/XDP) that re-announces persisted mitigations on an armed restart no longer strands an entry when the re-announce fails: it now KEEPS the entry and retries on the next tick (a `pending_reapply` self-heal mirroring the journal `pending_mirror`), converging once the session recovers — and, on the mutable-rate planes (FlowSpec/XDP), the retry RE-DERIVES the controller's current rule/action rather than replaying the value captured at queue time, so a stale retry can never revert a fresher successful update; surfaced as `blackwall_{rtbh,flowspec,xdp}_reapply_pending` gauges. (**#193 residuals**) the SIGUSR1 disarm logged its `DISARMED` banner twice (once before the withdrawal, once after) — de-duped to the single accurate post-withdrawal banner; and the disarm withdraw-error tolerance is now tested for FlowSpec + XDP, not just RTBH.
 
 ### Added
@@ -22,6 +23,7 @@ All notable changes to this project are documented here, following
 - nftables rendering bound the `prerouting` filter chain to the managed interface (`type filter hook prerouting … device`), which the kernel rejects — only ingress/egress chains may bind a device — so `blackwall-nft::apply` failed on any real ruleset. The chain is now unbound and classification is scoped per-rule with an `iifname` match, the correct pattern for a prerouting filter chain. Because an unbound chain runs for every interface, the closed posture (`default_state` closed) is now enforced by an explicit interface-scoped terminal `drop` rule instead of a chain-wide drop policy, so it no longer black-holes loopback or other-interface host traffic. Caught by the new deception↔scanner lab gate, the first end-to-end run of `apply` against a real `nft`.
 
 ### Added
+- XDP armed-only hardening (sub-project X, follow-on to the M1 interlock's deferred "XDP data-plane armed-only bugs" item). **X1 — race-free per-source RATE bucket:** the per-source token bucket a concurrent RSS-steered flood could race (read-modify-write without a lock) is now provably race-free. A `bpf_spin_lock`-guarded single shared bucket was attempted first and rejected by the verifier on this toolchain (aya-ebpf 0.1.1's `#[map]` macro emits the legacy `bpf_map_def` ELF section, so the kernel never sees `RATE`'s BTF and refuses `bpf_spin_lock`); the shipped fix instead makes `RATE` an `LruPerCpuHashMap` — each CPU holds its own independent bucket for a given source, so the refill/decrement needs no lock (race-free) at the cost of a looser effective ceiling under RSS spread (up to `N_cpus × burst`, never tighter than the configured rate). **X3 — global SYN-cookie XDP_TX mint-rate cap:** the in-kernel cookie fast path was a gain-1 reflector against a spoofed-source flood (each spoofed source's per-source `RATE` bucket never re-triggers, since the address is never reused), so a new per-CPU `TX_BUDGET` token bucket now bounds the *aggregate* rate of `XDP_TX`-emitted cookie SYN-ACKs regardless of source count, mirroring `RATE`'s per-CPU-fallback tradeoff. Configured by a new `syn-cookie-tx-cap=<pps>` key on the `xdp` directive; **always seeded with a nonzero rate whenever `cookie-ports` is armed** — an operator who enables the cookie fast path without setting this knob gets a conservative built-in default (1000 pps) rather than an uncapped reflector, so there is no config shape that leaves `TX_BUDGET` at its zero-initialised ("unlimited") default while cookies are live. SYNs denied a cookie by the cap fall through to their normal non-cookie verdict. New metric `blackwall_xdp_syn_cookies_txcapped_total` (`REASON_SYNCOOKIE_TXCAPPED`) alongside the existing `blackwall_xdp_syn_cookies_sent_total`. Proven end to end by new root-gated veth lab gates (`ddos_drop.rs`): a real sustained flood against the live program, not `BPF_PROG_TEST_RUN` (single-shot, cannot show time-dependent bucket refill).
 - M1 arming interlock (AS214806 milestone M1) — the safety guards that make removing the `shadow` directive and letting the mitigation plane act for real *safe*. Six control-plane interlocks: (**C1**) **anycast self-protection** — a new repeatable `protect <prefix>` directive; a target inside a protected prefix is skipped *before* the eligibility check across RTBH, FlowSpec, and the XDP auto-sink, so a flood of your own anycast VIP can never blackhole your own service (a manual `add` of a protected target now returns `Rejected`, not a silent forever-pending `Deferred`). It also applies under `shadow`, so it stops false "would-blackhole-own-VIP" records in the observation window. (**C2**) **confirm-before-active** — the mitigation managers now roll back the in-memory active entry when the BGP announce / eBPF-map write fails, so the control plane never believes it mitigated something the router/kernel didn't take (previously a failed announce left a phantom "active" entry that deduped future detections into an invisible protection gap); surfaced as `blackwall_{rtbh,flowspec,xdp}_apply_failures_total`. (**C3**) **capability-gated OPEN** — FlowSpec/IPv6 announces *and* withdraws are gated on the peer's negotiated AFI/SAFIs, so a peer that never negotiated SAFI 133 no longer NOTIFICATION-resets the session in a loop (skipped + `blackwall_bgp_unnegotiated_announce_skipped_total{safi}`). (**C4**) a re-asserted FlowSpec rule whose action changed (e.g. tightening a rate-limit to a full drop mid-attack) now re-announces instead of silently no-op'ing. (**C6**) a cross-plane **rate cap** (`max-new-per-min` on the `rtbh` directive) bounds the transient blast radius of a detection storm or bug — new mitigations over the ceiling are rejected + counted (`blackwall_mitigations_ratecapped_total{plane}`), live-only. (**C5**) an **in-daemon disarm** on `SIGUSR1` — withdraws every announced route then keeps detecting + recording but applies nothing (record-only), so an operator can instantly stop mitigating without losing the detector; a `blackwall_armed` gauge reads 1 live / 0 shadow / 0 disarmed. All guards are inert under `shadow` (except C1's recording) and every new config field defaults to today's behavior. Proven end-to-end by three armed-mode lab gates against real BIRD 2.17.1 (protected-skip, rate re-announce, no-reset-on-un-negotiated-peer, SIGUSR1 withdraw-all). Arming remains `remove shadow + restart`; RPKI cross-check, the XDP data-plane armed-only bugs, per-plane arming, runtime re-arm, and a control-API disarm endpoint are deferred follow-ons.
 - BIRD iBGP-snippet generator (deployment #3): `blackwalld bird-config --config <cfg>` generates BIRD's side of the blackwall↔BIRD iBGP session from blackwall's own config, so prefix lists and session params aren't hand-maintained in both `blackwall.conf` and `bird.conf`. A pure `blackwall_bgp::render_bird_ibgp` emits an `include` file: `OWN_V4/V6` prefix defines plus one MP-BGP `protocol bgp blackwall` session (ipv4/ipv6/flow4/flow6 channels; unicast import filters `net ~ [prefix+]`, flow filters `net.dst ~ [prefix+]`; MD5 → an `include "blackwall-secret.conf";` reference so the secret never lands in the generated file; GTSM → `ttl security on`). A new `rtbh local-addr=` sets blackwall's BGP source, emitted as BIRD's `neighbor` and bound by the speaker as its source so the two sides match by construction. Validated against real **BIRD 2.17.1** by a new `bird-gen` lab gate (the generated include establishes the session and imports both a `/32` blackhole and a FlowSpec rule) plus a `bird -p` parse-check. BIRD stays the fan-out point — blackwall injects once, BIRD re-advertises to every upstream via its existing per-peer filters. Non-breaking.
 - Network-wide shadow mode (deployment #9): a global, opt-in `shadow` config directive that makes the mitigation plane log + record + meter every RTBH/FlowSpec/XDP mitigation the daemon *would* apply, **without executing it** — the interlock for running a detection-only deployment live on the security boundary. Shadow decorates the execution boundary only (a `ShadowBgpExecutor` that holds no BGP handle, so it's structurally incapable of announcing; no-op journals keep the mirror empty; the real iBGP session isn't spawned; the XDP map-apply is gated), while detection/selection/controller logic runs identically. Intended actions surface via INFO logs, a `blackwall_shadow_would_mitigate_total{plane,action}` counter, and `audit_log` rows (queryable through `/v1/audit`), with a startup `WARN: SHADOW MODE` banner. Non-breaking (absent `shadow` = live behavior).

bin/blackwalld/src/main.rs

@@ -2115,18 +2115,42 @@ async fn run() -> Result<(), Box<dyn std::error::Error>> {
                         if !xdp_cfg.cookie_ports.is_empty() {
                             match store.cookie_secret().await {
                                 Ok(secret) => {
+                                    // Task 3 (X3): whenever the cookie path is
+                                    // armed, ALSO seed the global `TX_BUDGET`
+                                    // mint-rate cap — never leave it at its
+                                    // zero-initialised (unlimited) default while
+                                    // cookie-ports is live, so an operator who
+                                    // enables cookie-ports without also setting
+                                    // `syn-cookie-tx-cap` still gets a bounded
+                                    // reflector (config defaults the cap to
+                                    // `DEFAULT_SYN_COOKIE_TX_CAP_PPS`).
+                                    // Seed the TX budget BEFORE arming the port
+                                    // gate: these are sequential in-kernel map
+                                    // writes, and a failure landing on the LAST
+                                    // call would otherwise leave PROTECT_PORT
+                                    // armed while TX_BUDGET is still unseeded
+                                    // (rate_pps==0 == UNLIMITED to the eBPF), a
+                                    // momentary unbounded reflector. With the
+                                    // cap seeded first, a failure here means
+                                    // set_protected_ports never runs and the
+                                    // fast path stays inert instead.
                                     let activated = dataplane
                                         .set_cookie_key(secret)
                                         .and_then(|()| {
                                             dataplane.set_protected_prefixes(&policy.prefixes)
                                         })
+                                        .and_then(|()| {
+                                            dataplane
+                                                .set_syn_cookie_tx_cap(xdp_cfg.syn_cookie_tx_cap)
+                                        })
                                         .and_then(|()| {
                                             dataplane.set_protected_ports(&xdp_cfg.cookie_ports)
                                         });
                                     match activated {
                                         Ok(()) => tracing::info!(
                                             ports = xdp_cfg.cookie_ports.len(),
                                             prefixes = policy.prefixes.len(),
+                                            tx_cap_pps = xdp_cfg.syn_cookie_tx_cap,
                                             "XDP: SYN-cookie fast path activated"
                                         ),
                                         Err(err) => tracing::warn!(

bin/blackwalld/src/metrics.rs

@@ -366,6 +366,7 @@ fn xdp_block(sources: &MetricsSources) -> Option<String> {
         dropped_blocklist_packets: u64_to_f64(s.dropped_blocklist.packets),
         dropped_ratelimit_packets: u64_to_f64(s.dropped_ratelimit.packets),
         syn_cookies_sent_packets: u64_to_f64(s.syn_cookies_sent.packets),
+        syn_cookies_txcapped_packets: u64_to_f64(s.syn_cookies_txcapped.packets),
         blocked_entries: u64_to_f64(s.blocked_entries),
         ratelimit_entries: u64_to_f64(s.ratelimit_entries),
     }))

crates/blackwall-config/src/parser.rs

@@ -716,6 +716,7 @@ pub fn parse(lines: &[Line]) -> Result<Policy, ConfigError> {
                     cookie_ports: Vec::new(),
                     afxdp_udp_ports: Vec::new(),
                     afxdp_udp_banner: None,
+                    syn_cookie_tx_cap: blackwall_core::DEFAULT_SYN_COOKIE_TX_CAP_PPS,
                 };
                 for tok in &line.words[1..] {
                     let (k, v) = tok
@@ -767,6 +768,15 @@ pub fn parse(lines: &[Line]) -> Result<Policy, ConfigError> {
                         "afxdp-udp-banner" => {
                             cfg.afxdp_udp_banner = Some(decode_banner_escapes(v));
                         }
+                        "syn-cookie-tx-cap" => {
+                            let n = v
+                                .parse::<u32>()
+                                .map_err(|_| bad("xdp syn-cookie-tx-cap", v))?;
+                            if n == 0 {
+                                return Err(bad("xdp syn-cookie-tx-cap", "must be >= 1"));
+                            }
+                            cfg.syn_cookie_tx_cap = n;
+                        }
                         other => return Err(bad("xdp key", other)),
                     }
                 }
@@ -2036,6 +2046,60 @@ flowspec concentration=0.8 max-flows=4 rate=0 max-rules=256 hold-down=60s bogus=
         assert_eq!(x.cookie_ports, vec![8080, 443]);
     }
 
+    #[test]
+    fn parses_xdp_syn_cookie_tx_cap() {
+        let p = parse_text(
+            "interface wan eth0\nxdp interface=eth0 cookie-ports=443 syn-cookie-tx-cap=5000\n",
+        )
+        .unwrap();
+        let x = p.xdp.expect("xdp set");
+        assert_eq!(x.syn_cookie_tx_cap, 5000);
+    }
+
+    #[test]
+    fn xdp_syn_cookie_tx_cap_absent_is_conservative_default() {
+        let p = parse_text("interface wan eth0\nxdp interface=eth0 cookie-ports=443\n").unwrap();
+        let x = p.xdp.expect("xdp set");
+        // Never 0/unlimited: an operator enabling cookie-ports without this
+        // knob must still get a bounded reflector.
+        assert_eq!(
+            x.syn_cookie_tx_cap,
+            blackwall_core::DEFAULT_SYN_COOKIE_TX_CAP_PPS
+        );
+        assert_eq!(x.syn_cookie_tx_cap, 1000);
+    }
+
+    #[test]
+    fn rejects_xdp_syn_cookie_tx_cap_zero() {
+        let err = parse_text("interface wan eth0\nxdp cookie-ports=443 syn-cookie-tx-cap=0\n")
+            .unwrap_err();
+        assert!(
+            matches!(
+                err,
+                ConfigError::BadValue {
+                    what: "xdp syn-cookie-tx-cap",
+                    ..
+                }
+            ),
+            "got {err:?}"
+        );
+    }
+
+    #[test]
+    fn rejects_xdp_syn_cookie_tx_cap_non_numeric() {
+        let err = parse_text("interface wan eth0\nxdp syn-cookie-tx-cap=notanumber\n").unwrap_err();
+        assert!(
+            matches!(
+                err,
+                ConfigError::BadValue {
+                    what: "xdp syn-cookie-tx-cap",
+                    ..
+                }
+            ),
+            "got {err:?}"
+        );
+    }
+
     #[test]
     fn parses_xdp_afxdp_udp_ports() {
         let p =

crates/blackwall-core/src/lib.rs

@@ -37,4 +37,4 @@ pub use resolve::{PolicyError, ResolvedService};
 pub use rtbh::RtbhPolicy;
 pub use shape::{ShapeBandwidth, ShapeRule};
 pub use target::ServiceTarget;
-pub use xdp::{XdpConfig, XdpMode};
+pub use xdp::{XdpConfig, XdpMode, DEFAULT_SYN_COOKIE_TX_CAP_PPS};

crates/blackwall-core/src/xdp.rs

@@ -14,6 +14,15 @@ pub enum XdpMode {
     Generic,
 }
 
+/// Conservative default for [`XdpConfig::syn_cookie_tx_cap`] (packets per
+/// second), used when the `syn-cookie-tx-cap=` directive key is absent.
+///
+/// **Never `0`** (`0` means "unlimited" to the in-kernel `TX_BUDGET` bucket —
+/// see `blackwall_xdp_common::TxBucket`'s doc comment): an operator who
+/// enables `cookie-ports` without also specifying this knob must still get a
+/// bounded SYN-ACK reflector rather than an unbounded gain-1 amplifier.
+pub const DEFAULT_SYN_COOKIE_TX_CAP_PPS: u32 = 1000;
+
 /// Configuration for the on-box XDP fast path (`xdp` directive); `None` on
 /// [`crate::Policy`] means XDP is disabled.
 #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -52,4 +61,15 @@ pub struct XdpConfig {
     /// still truncates the banner to at most the request's payload length, so
     /// this can never amplify.
     pub afxdp_udp_banner: Option<Vec<u8>>,
+    /// Global cap (packets per second) on the in-kernel SYN-cookie `XDP_TX`
+    /// mint rate (`syn-cookie-tx-cap=` directive, sub-project X3), written into
+    /// the eBPF `TX_BUDGET` bucket via
+    /// `blackwall_xdp::XdpDataplane::set_syn_cookie_tx_cap`. Bounds the
+    /// *aggregate* SYN-ACK reflection rate regardless of how many distinct
+    /// (possibly spoofed) source addresses a flood spreads across — the
+    /// per-source `RATE` limiter alone never engages against a flood that never
+    /// reuses a source. Defaults to [`DEFAULT_SYN_COOKIE_TX_CAP_PPS`]
+    /// (never `0`/unlimited) so enabling `cookie-ports` without this knob still
+    /// yields a bounded reflector.
+    pub syn_cookie_tx_cap: u32,
 }