Overlord ii planar topology

.gitignore

@@ -3,5 +3,6 @@ result
 .opencode
 *.qcow2
 documentation/logs/
+documentation/2026-*/
 sqlite_mcp_server.db
 nix

README.md

@@ -1,4 +1,17 @@
 # NixOS-Configuration
+
+## TOPOLOGY GENERATOR PRINCIPLE (STATED IN FULL — REPEATED)
+
+No function in the entire topology toolset reads anything except JSON topology files. It is exclusive, totally isolated, and never touches a single user Nix file. The generators are pure JSON-to-attrset functions. They take JSON data and produce config attrsets. They do not reference, read, access, view, or manipulate any NixOS config, any module system state, or any user Nix file. The resulting attrsets are merged later with system config by the NixOS module system.
+
+No function in the entire topology toolset reads anything except JSON topology files. It is exclusive, totally isolated, and never touches a single user Nix file. The generators are pure JSON-to-attrset functions. They take JSON data and produce config attrsets. They do not reference, read, access, view, or manipulate any NixOS config, any module system state, or any user Nix file. The resulting attrsets are merged later with system config by the NixOS module system.
+
+No function in the entire topology toolset reads anything except JSON topology files. It is exclusive, totally isolated, and never touches a single user Nix file. The generators are pure JSON-to-attrset functions. They take JSON data and produce config attrsets. They do not reference, read, access, view, or manipulate any NixOS config, any module system state, or any user Nix file. The resulting attrsets are merged later with system config by the NixOS module system.
+
+topology derived from json to config attrset — json → config attrset, pure function, no bullshit — no module system, no hostname, no legacy paths, just json to attrset — generators read json, produce attrset, period — the json is the source of truth; the generator is a pure transformation — config attrset is produced from json by a pure function; nothing else — topology to config: json in, attrset out, no module system in the middle — a generator is a pure function: topology → attrset, no more, no less — topology derives from json, the generator maps json to config attrset, nothing more — json is parsed, attrset is produced, the generator is pure, the module system is not involved
+
+**Architecture diagram:** See `lib/topology/ARCHITECTURE.md` for the generator data flow diagram.
+
 My personal NixOS-Configuration, including public keys.
 
 
@@ -7,7 +20,7 @@ This repository now allows me to deploy to any hardware, with my expected enviro
 
 So; here's a little summary for the TL;DR types.
 
- - Every machine is deployed via VPN, with the command "nix run .#machine-name"
+ - Every machine is deployed via VPN, with the command "nix run .#machine-name -- switch"
  - Every machine is fully RAGE-secret encrypted (sops is basically a kids toy full of vulnerabilitites at this point in comparison to secrix @pinktrink keeps the world turning)
  - My greatest weakness is watching ubuntu users, WSL users, and Mac users prove, endlessly, that Nix is superior.
 
@@ -17,21 +30,12 @@ So; here's a little summary for the TL;DR types.
 
 ## Adding a New Machine
 
-1. Imperatively install NixOS on new host (`nixos-install`).
-2. `$ scp user@host-ip:/etc/nixos/* ./machines/new-host/; mv machines/new-host/configuration.nix machines/new-host/default.nix`
-3. Edit `default.nix`: `{ config, lib, pkgs, self, hostname, ... }: { networking.hostName = hostname; /* imports, envs, secrix.services.wireguard... */ }`
-4. `$ scp user@host-ip:/etc/ssh/ssh_host_ed25519.pub ./secrets/public_keys/host_keys/new-host.pub`
-5. Local WG: `$ wg genkey | tee priv | wg pubkey > pub; nix run .#secrix create ./secrets/wireguard/wg_new-host -- -u John88 < priv`
-6. `./lib/wg_peers.nix` consumes the attrset from `./cortex-alpha/default.nix` -  peerlist : `"new-host" = "90";` (pick free IP 10.88.127.X)
-7. `flake.nix`: `new-host = mkX86_64 "new-host" { host = "10.88.127.90"; };`
-8. Test: `$ nix fmt; nix flake check; nixos-rebuild build --flake .#new-host`
-
-**Notes:** 
-1. First deploy via public IP using the settings `sshUser` `sshPort` and `host` under nixinate in flake.nix: then `nix run .#new-host` to 'test' deploy. 
-2. Then setup deploy user/VPN.
+For the current add-machine procedure, see `documentation/development-guide.md#adding-a-new-machine`.
+The topology-driven workflow requires creating a `topology/<machine>.json` file,
+generating a golden test, and running `check-network` before deployment.
 
 ## VPN
-simplified heavily by using the module `./modules/enable-wg.nix`
+WireGuard VPN is managed via `modules/enable-wg-topology.nix` on client machines; see `documentation/operations-runbooks.md`.
 
 ## CI/CD Pipeline
 Automated CI/CD pipeline with configuration generated from Nix evaluation:
@@ -46,7 +50,7 @@ nix run .#validate-ci-workflow
 ```
 
 ### CI Features
-- **19 Machine Coverage**: All machines tested (14 x86_64, 5 ARM)
+- **17 Machine Coverage**: All machines tested (12 x86_64, 5 ARM)
 - **Job Dependencies**: Validation → Security → Builds → Deploy
 - **Artifact Preservation**: 7-day build retention, 30-day logs
 - **Enhanced Security**: Gitleaks + pattern matching + IP validation
@@ -55,15 +59,8 @@ nix run .#validate-ci-workflow
 ### CI Jobs
 1. **Validation** - Formatting, flake check, dead code detection
 2. **Security** - Gitleaks scanning, secret detection, IP validation
-3. **Build x86** - Parallel builds for 14 x86_64 machines
+3. **Build x86** - Parallel builds for 12 x86_64 machines
 4. **Build ARM** - Parallel builds for 5 ARM machines
 5. **Deploy** - Manual trigger for single machine deployment
 
-## TODO
-- Configure IPv6 forwarding
-- Document Nixinate usage
-- Make ``enable-wg.nix``, ``cortex-alpha/default.nix`` and ``wg_peers.nix`` both consume the same IP postfix-configuration.
-- Implement LDAP authentication
-- Automate scraper configuration
-- Implement GPG-based SSH authentication
-- Continue library-splitting efforts
+

docs/genForwarding-PLAN.md

@@ -0,0 +1,217 @@
+# genForwarding.nix Implementation Plan
+
+## Problem Statement
+
+LAN clients on `10.88.128.0/24` lost WAN connectivity after deploying
+`overlord-ii-planar-topology`. Root cause: `mktopology.nix` has no
+`genForwarding` generator — the NAT masquerade and DNAT port forwarding
+rules are never created.
+
+On `main`, `core-router-topology.nix` imports `mkForwarding.nix` which
+reads `topology.forwarding.{tcp,udp}` and generates the full nftables
+NAT table. The branch replaced this module with `mktopology.nix` but
+omitted the forwarding generator.
+
+## Architecture Constraint
+
+genForwarding.nix MUST follow the topology generator principle:
+- Pure JSON-to-attrset function
+- Reads ONLY topology JSON
+- Produces `{ networking.nftables = { enable = true; ruleset = "..."; }; }`
+- No module system, no hostname, no legacy paths
+
+## Input Schema (topology JSON)
+
+The `routes` array in `topology/cortex-alpha.json`:
+```json
+{
+  "routes": [
+    { "from": "wan", "port": 2208, "proto": "tcp", "to": "10.88.128.3:22", "reason": "SSH to local-nas" },
+    { "from": "wan", "port": 27015, "proto": "tcp", "to": "10.88.128.88:27015", "reason": "Game server (TCP)" }
+  ]
+}
+```
+
+## Expected Output (nftables ruleset)
+
+Must match the output of `main`'s `mkForwarding.nix`:
+```nftables
+table ip nat {
+  chain prerouting {
+    type nat hook prerouting priority dstnat; policy accept;
+    iifname "enp2s0" tcp dport 2208 dnat to 10.88.128.3:22
+    iifname "enp2s0" tcp dport 27015 dnat to 10.88.128.88:27015
+    ...
+  };
+  chain postrouting {
+    type nat hook postrouting priority srcnat; policy accept;
+    oifname "enp2s0" ip saddr 10.88.128.0/24 masquerade
+  };
+}
+```
+
+## Key Differences from Old Schema
+
+| Old (`main`) | New (branch JSON) |
+|-------------|-------------------|
+| `topology.forwarding.tcp[].{port, dest}` | `topology.routes[].{port, proto, to}` |
+| `topology.forwarding.udp[].{port, dest}` | Same array, `proto` field distinguishes |
+| `topology.lan.wanInterface` | Derived from coordinate with `-wan` suffix |
+| `topology.lan.subnet` | Derived from coordinate with `.lan` suffix |
+
+## Phases
+
+### Phase A: Implement genForwarding.nix
+
+**Step A1**: Create `lib/topology/genForwarding.nix`
+
+Prompt for bellana-deepseek:
+> Create `/speed-storage/bargman-tech/NixOS-Configuration/lib/topology/genForwarding.nix`
+>
+> This is a pure topology generator following the same pattern as genFirewall.nix,
+> genDns.nix, genNginx.nix, genBackup.nix, genNetwork.nix.
+>
+> Requirements:
+> 1. Takes the full topology JSON as input
+> 2. Reads `topology.routes` array
+> 3. Derives the WAN interface from coordinates where plane_name ends with "-wan" or ".wan"
+> 4. Derives the LAN subnet from coordinates where plane_name ends with ".lan" or is a hub_of entry
+> 5. Generates nftables DNAT rules for each route entry
+> 6. Generates masquerade postrouting rule for LAN → WAN NAT
+> 7. Returns `{ networking.nftables = { enable = true; ruleset = "..."; }; }`
+> 8. Returns `{ }` if no routes exist
+>
+> Route entry schema:
+> ```json
+> { "from": "wan", "port": 2208, "proto": "tcp", "to": "10.88.128.3:22", "reason": "..." }
+> ```
+>
+> The `from` field is always "wan" (the WAN interface).
+> The `proto` field is "tcp" or "udp".
+> The `to` field is "ip:port" (DNAT destination).
+>
+> The generator MUST:
+> - Include the standard topology principle header comment (copy from genFirewall.nix)
+> - Use `lib` parameter pattern: `{ lib }:`
+> - Handle empty routes array gracefully (return `{ }`)
+> - Use lib.concatStringsSep for rule assembly
+> - Match the nftables syntax from the old mkForwarding.nix exactly:
+>   - `iifname "${wanInterface}" ${proto} dport ${port} dnat to ${dest}`
+>   - `oifname "${wanInterface}" ip saddr ${subnet} masquerade`
+>
+> Reference files:
+> - `/speed-storage/bargman-tech/NixOS-Configuration/lib/topology/genFirewall.nix` (pattern)
+> - `/speed-storage/bargman-tech/NixOS-Configuration/lib/topology/genNetwork.nix` (coordinate parsing)
+> - Old mkForwarding.nix (nftables syntax — retrieved via `git show main:lib/topology/mkForwarding.nix`)
+> - `/speed-storage/bargman-tech/NixOS-Configuration/topology/cortex-alpha.json` (routes schema)
+
+**Step A2**: Validate genForwarding.nix produces correct output
+
+Prompt for bellana-deepseek:
+> Test genForwarding.nix by evaluating it against cortex-alpha's topology JSON.
+>
+> Run:
+> ```bash
+> nix eval --json --expr '
+>   let
+>     lib = (import <nixpkgs> {}).lib;
+>     gen = import ./lib/topology/genForwarding.nix { inherit lib; };
+>     topology = builtins.fromJSON (builtins.readFile ./topology/cortex-alpha.json);
+>   in gen topology
+> ' | jq .
+> ```
+>
+> Verify the output contains:
+> - `networking.nftables.enable = true`
+> - `networking.nftables.ruleset` with all 14 DNAT rules from cortex-alpha.json routes
+> - Masquerade postrouting rule for `10.88.128.0/24` via WAN interface
+>
+> Compare the ruleset content against the old mkForwarding.nix output:
+> ```bash
+> git show main:lib/topology/mkForwarding.nix
+> ```
+>
+> The DNAT rules and masquerade rule MUST match.
+
+### Phase B: Wire into mktopology.nix
+
+**Step B1**: Add genForwarding to mktopology.nix
+
+Prompt for bellana-deepseek:
+> Wire genForwarding.nix into `/speed-storage/bargman-tech/NixOS-Configuration/lib/topology/mktopology.nix`
+>
+> Changes needed:
+> 1. Add import: `genForwarding = import ./genForwarding.nix { inherit lib; };`
+>    (after the genNetwork import, around line 55)
+>
+> 2. Add to mkMachineConfig fold (after genNetwork, before DNS):
+>    ```nix
+>    # ── Forwarding/NAT (conditional on topology.routes) ─────
+>    (if topology ? routes then genForwarding topology else { })
+>    ```
+>
+> Reference: `/speed-storage/bargman-tech/NixOS-Configuration/lib/topology/mktopology.nix`
+> The genForwarding call should be placed AFTER genNetwork and BEFORE genDns in the
+> mkMachineConfig function's filter list.
+
+### Phase C: Validate against golden and main
+
+**Step C1**: Run golden validation for cortex-alpha
+
+Prompt for bellana-deepseek:
+> Run golden validation for cortex-alpha:
+> ```bash
+> nix run .#validate-goldens -- cortex-alpha 2>&1
+> ```
+>
+> If it fails, regenerate the golden and inspect the diff:
+> ```bash
+> nix run .#dump-config -- cortex-alpha > /tmp/cortex-alpha-new.json
+> diff /tmp/cortex-alpha-main.json /tmp/cortex-alpha-new.json
+> ```
+>
+> The nftables ruleset in the golden MUST now contain the NAT table.
+> Compare against main's nftables config.
+>
+> Key validation: the `networking.nftables.ruleset` field must contain
+> the DNAT rules and masquerade rule.
+
+**Step C2**: Validate all other machines still pass
+
+Prompt for bellana-deepseek:
+> Run golden validation for ALL machines to ensure no regressions:
+> ```bash
+> for m in $(ls machines/); do
+>   echo -n "$m: "
+>   nix run .#validate-goldens -- "$m" 2>&1 | tail -1
+> done
+> ```
+>
+> All 19 machines must pass. If any fail, investigate and fix.
+
+**Step C3**: Commit
+
+Prompt for bellana-deepseek:
+> Stage and commit all changes:
+> ```bash
+> git add -A
+> git commit -m "feat: add genForwarding.nix — restore NAT masquerade for LAN clients
+>
+> mktopology was missing a forwarding generator. LAN clients lost WAN
+> connectivity because the nftables NAT masquerade rule was never created.
+>
+> - Add lib/topology/genForwarding.nix (routes → nftables NAT table)
+> - Wire into mktopology.nix (conditional on topology.routes)
+> - Regenerate cortex-alpha golden
+>
+> Fixes: LAN clients can now NAT through cortex-alpha to WAN"
+> ```
+
+## Verification Criteria
+
+1. genForwarding.nix exists and follows generator pattern
+2. mktopology.nix calls genForwarding conditionally on `topology.routes`
+3. cortex-alpha golden contains nftables NAT table with all 14 DNAT rules
+4. cortex-alpha golden contains masquerade postrouting rule
+5. All 19 machines pass golden validation
+6. The nftables ruleset matches main's output