diff --git a/CHANGELOG.md b/CHANGELOG.md index eb5c29cd..b956f2a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ All notable changes to this project are documented in this file. The format is b - **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197). - **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202). - **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201). +- **Opt-in Linux transparent TCP inbound & raw TCP recording (P4.1b)**: on Linux when traffic capture is enabled, setting `ARTEX_TRANSPARENT_TCP_ENABLED=1` (loopback `127.0.0.1:1081` by default) or `ARTEX_TRANSPARENT_TCP_ADDR=` starts an IPv4 transparent TCP listener fed by operator-managed `iptables`/`nftables` `REDIRECT` rules (off by default; ARTEX never touches the host firewall; root/`CAP_NET_ADMIN` is required only for the operator's redirect rules; IPv6 is not covered). Recovers original IPv4 destinations via `SO_ORIGINAL_DST` with self-redirect loop protection, reuses the P4.1a target gate (`intercept.Interceptor.DialSOCKS5`), egress governor (`egress.Do`), half-close relay bounds, and SQLite `raw_tcp_streams` store (atomic `_blobs` spill, no HTTP `exchanges` row, no PostgreSQL migration). See `docs/traffic-transparent.md`. - **Opt-in SOCKS5 inbound proxy & raw TCP recording (P4.1a)**: when traffic capture is enabled, setting `ARTEX_SOCKS5_ENABLED=1` (loopback `127.0.0.1:1080` by default) or `ARTEX_SOCKS5_ADDR=` starts an RFC 1928 SOCKS5 `CONNECT` listener alongside the HTTP recording proxy (off by default; HTTP proxy and agent proxy env vars are unchanged; listener does not start if traffic recording is disabled). Accepts loopback no-auth and RFC 1929 username/password without verifying credentials (carrying `artex-pace` in the username; binding to a non-loopback address therefore exposes an unauthenticated proxy, though the target gate and governor still apply), rejects `BIND`/`UDP ASSOCIATE` with `0x07`, enforces the existing target gate (`intercept.Interceptor.DialSOCKS5` / `intercept.PinnedDialer`) and egress governor (`egress.Do`) before dialing, relays half-close-aware TCP with concurrency/byte/idle/handshake bounds, and records one row per connection in SQLite `raw_tcp_streams` with capped directional previews and atomic (temp+rename) `_blobs` refs without writing HTTP `exchanges` rows. See `docs/traffic-socks5.md`. - **Opt-in worker cache-prefix split (P3.5 Phase C)**: setting `ARTEX_CACHE_PREFIX=1` (off by default) splits the `worker` system prompt at `DynamicBoundary = 1` into a sibling-shareable static prefix (`body + untrustedContentTail + workerTrafficBlock`) and a per-intent dynamic suffix (`workerArtifactSpec(runDir)` through ``) without reordering sections or altering the visible joined prompt text. Planner, mainagent, chat, and goals are unchanged; custom worker prompts referencing `{{.Now}}` remain volatile in the static segment. See `docs/context-telemetry.md`. - **Opt-in LLM context size telemetry (P3.5 Phase A)**: setting `ARTEX_CONTEXT_TELEMETRY=1` (off by default) records per-call request section byte sizes (`system_static`/`system_dynamic` split at `DynamicBoundary`, serialized `tools`, `messages` by role, `tool_result` subset, and `last_message_bytes`), tool/message counts, and token/cache usage in the new `llm_context` table (migration `0009_llm_context.sql`), with zero prompt/behavior change and zero content stored, independent of `llm_record`. `GET /api/llm/context/summary` aggregates input percentiles (`p50`/`p90`/`p99`), pro-rata section shares over real `input_tokens`, `cache_read/input` ratios by `(worker, model)`, and top tasks. See `docs/context-telemetry.md`. diff --git a/docs/traffic-transparent.md b/docs/traffic-transparent.md new file mode 100644 index 00000000..8648eabd --- /dev/null +++ b/docs/traffic-transparent.md @@ -0,0 +1,81 @@ +# Opt-in Linux Transparent TCP Inbound & Raw TCP Recording (P4.1b) + +ARTEX ships an opt-in **Linux-only IPv4 transparent TCP inbound listener** in `traffic/` that accepts connections redirected by `iptables` or `nftables` `REDIRECT` rules, recovers each socket's original IPv4 destination via `getsockopt(SOL_IP, SO_ORIGINAL_DST)`, enforces the existing target gate and egress governor before dialing, and records the connection in SQLite `raw_tcp_streams`. + +## Opt-in Configuration (Off by Default) + +| Environment Variable | Default | Behavior | +|---|---|---| +| `ARTEX_TRANSPARENT_TCP_ENABLED` | unset (off) | Set to `1` / `true` / `on` / `yes` to start the transparent TCP listener on `127.0.0.1:1081` (IPv4 loopback). Explicit `0` / `false` / `off` / `no` forces it off even if `ARTEX_TRANSPARENT_TCP_ADDR` is set. | +| `ARTEX_TRANSPARENT_TCP_ADDR` | `127.0.0.1:1081` | Explicit IPv4 TCP bind address; setting a non-empty value also enables the listener unless `ARTEX_TRANSPARENT_TCP_ENABLED` is explicitly false. Use `0.0.0.0:1081` only when redirecting traffic from other interfaces (`PREROUTING`) and protect the port with a host firewall. | + +- **Flag off**: no transparent TCP listener is started, no firewall rules are touched, and the HTTP MITM proxy is unchanged. +- **Requires traffic recorder**: the server only starts the transparent TCP listener when the traffic recorder is enabled (inside the traffic proxy block in `server/manager.go`). If traffic capture is disabled, setting these environment variables does not open a listener. +- **Linux only**: on non-Linux operating systems (`windows`, `darwin`), leaving the flag off is a no-op; explicitly enabling it logs `ErrTransparentTCPNotLinux` and leaves the listener disabled. +- **IPv4 only (IPv6 not covered)**: P4.1b recovers `SO_ORIGINAL_DST` (`SOL_IP`, `AF_INET`) for IPv4 TCP sockets only. IPv6 (`IP6T_SO_ORIGINAL_DST`) is **not covered**; IPv6 bind addresses and IPv6 sockets are refused. +- **Firewall never touched by ARTEX**: ARTEX never installs, modifies, or deletes `iptables`/`nftables` rules. Root or `CAP_NET_ADMIN` is required **only** by the operator when installing and cleaning up the host redirect rules—never to run ARTEX itself. + +## Firewall Rules & Cleanup (Operator-Managed) + +Choose **either** `iptables` **or** `nftables`. Always exclude the UID running ARTEX (example: `artex`) from local `OUTPUT` redirect rules so ARTEX's own outbound dials do not loop back into the transparent listener. + +### Option 1: `iptables` (IPv4 local `OUTPUT` REDIRECT) + +Install into a dedicated chain so cleanup is atomic: + +```bash +# Requires root or CAP_NET_ADMIN +sudo iptables -t nat -N ARTEX_TRANSPARENT +# Never redirect ARTEX's own outbound dials or loopback traffic +sudo iptables -t nat -A ARTEX_TRANSPARENT -m owner --uid-owner artex -j RETURN +sudo iptables -t nat -A ARTEX_TRANSPARENT -d 127.0.0.0/8 -j RETURN +# Example: redirect outbound TCP to lab target 198.51.100.0/24 into ARTEX on 127.0.0.1:1081 +sudo iptables -t nat -A ARTEX_TRANSPARENT -p tcp -d 198.51.100.0/24 -j REDIRECT --to-ports 1081 +sudo iptables -t nat -A OUTPUT -p tcp -j ARTEX_TRANSPARENT +``` + +Cleanup (exact removal): + +```bash +sudo iptables -t nat -D OUTPUT -p tcp -j ARTEX_TRANSPARENT +sudo iptables -t nat -F ARTEX_TRANSPARENT +sudo iptables -t nat -X ARTEX_TRANSPARENT +``` + +*(For traffic arriving on another interface—for example a container or routed VM—bind `ARTEX_TRANSPARENT_TCP_ADDR=0.0.0.0:1081`, attach `ARTEX_TRANSPARENT` to `PREROUTING` instead of `OUTPUT`, and drop direct non-redirected packets to port 1081 at the host firewall.)* + +### Option 2: `nftables` (IPv4 local `OUTPUT` REDIRECT) + +Install into a dedicated table so cleanup is a single command: + +```bash +# Requires root or CAP_NET_ADMIN +sudo nft -f - <<'NFT' +table ip artex_transparent { + chain output { + type nat hook output priority -100; policy accept; + meta skuid "artex" return + ip daddr 127.0.0.0/8 return + ip daddr 198.51.100.0/24 tcp redirect to :1081 + } +} +NFT +``` + +Cleanup (exact removal): + +```bash +sudo nft delete table ip artex_transparent +``` + +## Policy Enforcement, Loop Guard & Storage (Shared with P4.1a) + +For every accepted TCP connection: + +1. **Original destination recovery**: reads `getsockopt(SOL_IP, SO_ORIGINAL_DST)` under `HandshakeTimeout` (default `10s`), parses `struct sockaddr_in` (rejecting short buffers, non-`AF_INET` families including IPv6, port `0`, and `0.0.0.0`), and rejects **self-redirects** (direct non-redirected connections where `SO_ORIGINAL_DST` equals the listener's own address/port) before dialing. +2. **Egress governor (`egress.Do(egress.Current(), ...)`)**: enforces the active noise profile (`PaceTimeout`, default `30s`) before dialing. +3. **Target gate (`intercept.Interceptor.DialSOCKS5` / `intercept.PinnedDialer`)**: enforces global asset block rules and any enabled `intercept.TargetScope` pins/exemptions before opening any outbound socket. Because the transparent listener only knows the recovered destination IP (never a DNS hostname), hostname-based block or allow rules cannot match—IP and CIDR rules are the ones that apply. Refused connections are closed immediately without dialing and without writing a store row. +4. **Relay & bounds**: reuses the P4.1a full-duplex half-close-aware relay (`CloseWrite`) and defaults (`MaxConcurrent=64`, `MaxBytesPerDir=4 MiB`, `PreviewBytes=8 KiB`, `IdleTimeout=2m`, `HandshakeTimeout=10s`, `DialTimeout=30s`, `PaceTimeout=30s`). +5. **SQLite storage**: records one row per completed connection in `raw_tcp_streams` (`protocol='tcp'`, atomic temp+rename `_blobs/sha256/` spill + `blob_refs`), with **zero HTTP `exchanges` rows** and **no PostgreSQL migration**. + +Closing `Traffic` stops accepting new transparent connections, closes active sockets, and waits for all handler goroutines to exit before closing SQLite. diff --git a/server/manager.go b/server/manager.go index de1a2945..47c486c8 100644 --- a/server/manager.go +++ b/server/manager.go @@ -471,6 +471,14 @@ func NewManager(dir, proxyAddr string) (*Manager, error) { log.Printf("[traffic/socks5] opt-in SOCKS5 CONNECT listener on %s", bound) } } + if tcfg, ok := traffic.TransparentTCPConfigFromEnv(); ok { + tcfg.DialContext = m.interceptor.DialSOCKS5 + if bound, terr := tr.StartTransparentTCP(tcfg); terr != nil { + log.Printf("[traffic/transparent] disabled: %v", terr) + } else { + log.Printf("[traffic/transparent] opt-in transparent TCP listener on %s (Linux IPv4 SO_ORIGINAL_DST)", bound) + } + } go func() { log.Printf("[traffic] recording proxy on %s (set HTTP_PROXY=%s + trust _ca CA)", proxyAddr, tr.ProxyAddr()) if err := tr.Start(); err != nil { diff --git a/server/transparent_test.go b/server/transparent_test.go new file mode 100644 index 00000000..fea8e40f --- /dev/null +++ b/server/transparent_test.go @@ -0,0 +1,48 @@ +package server + +import ( + "runtime" + "testing" + + "github.com/Autumn-27/artex/db" + "github.com/Autumn-27/artex/traffic" +) + +func TestManagerTransparentTCPOptInFlag(t *testing.T) { + // 1. Flag off (default): Manager starts NO transparent TCP listener. + t.Setenv(traffic.TransparentTCPEnabledEnv, "") + t.Setenv(traffic.TransparentTCPAddrEnv, "") + + mOff, err := NewManager(t.TempDir(), "127.0.0.1:0") + if err != nil { + db.SkipPGUnavailable(t, err) + } + defer mOff.Close() + if mOff.traffic == nil { + t.Fatal("expected traffic recorder to be initialized") + } + if got := mOff.traffic.TransparentTCPAddr(); got != "" { + t.Fatalf("TransparentTCPAddr with flag off = %q, want empty", got) + } + + // 2. Flag on (ARTEX_TRANSPARENT_TCP_ADDR=127.0.0.1:0): starts on Linux, + // stays cleanly disabled on non-Linux without crashing NewManager. + t.Setenv(traffic.TransparentTCPAddrEnv, "127.0.0.1:0") + + mOn, err := NewManager(t.TempDir(), "127.0.0.1:0") + if err != nil { + db.SkipPGUnavailable(t, err) + } + defer mOn.Close() + + gotAddr := mOn.traffic.TransparentTCPAddr() + if runtime.GOOS == "linux" { + if gotAddr == "" { + t.Fatal("expected transparent TCP listener on Linux when ARTEX_TRANSPARENT_TCP_ADDR is set") + } + } else { + if gotAddr != "" { + t.Fatalf("expected transparent TCP listener disabled on %s, got %q", runtime.GOOS, gotAddr) + } + } +} diff --git a/traffic/traffic.go b/traffic/traffic.go index a6132b91..e0379c30 100644 --- a/traffic/traffic.go +++ b/traffic/traffic.go @@ -258,6 +258,10 @@ type Traffic struct { // socks holds the opt-in SOCKS5 CONNECT listener when started via StartSOCKS5. socksMu sync.Mutex socks *socks5Server + // transparent holds the opt-in Linux transparent TCP listener when started + // via StartTransparentTCP (P4.1b). + transparentMu sync.Mutex + transparent *transparentServer } // Open initializes the traffic tree, blob store and SQLite index under dir. @@ -478,6 +482,13 @@ func (t *Traffic) Close() error { if srv != nil { _ = srv.Close() } + t.transparentMu.Lock() + tsrv := t.transparent + t.transparent = nil + t.transparentMu.Unlock() + if tsrv != nil { + _ = tsrv.Close() + } t.reaping.Wait() t.sweeping.Wait() return t.db.Close() diff --git a/traffic/transparent.go b/traffic/transparent.go new file mode 100644 index 00000000..5609ad96 --- /dev/null +++ b/traffic/transparent.go @@ -0,0 +1,460 @@ +package traffic + +import ( + "context" + "encoding/binary" + "errors" + "fmt" + "log" + "net" + "net/url" + "os" + "strconv" + "strings" + "sync" + "time" + + "github.com/Autumn-27/artex/egress" +) + +// Opt-in environment flags for the Linux transparent TCP inbound listener (P4.1b). +// Off by default: when neither is set, no transparent listener is started, no +// firewall rules are touched, and the HTTP proxy is unchanged. +const ( + // TransparentTCPEnabledEnv turns on the Linux transparent TCP listener on + // DefaultTransparentTCPAddr when set to a truthy value (1/true/yes/on). + TransparentTCPEnabledEnv = "ARTEX_TRANSPARENT_TCP_ENABLED" + // TransparentTCPAddrEnv optionally overrides the IPv4 bind address + // (default 127.0.0.1:1081). Setting a non-empty address also enables the + // listener unless ARTEX_TRANSPARENT_TCP_ENABLED is explicitly false/0/off/no. + TransparentTCPAddrEnv = "ARTEX_TRANSPARENT_TCP_ADDR" + + DefaultTransparentTCPAddr = "127.0.0.1:1081" + + // afInet matches POSIX/Linux AF_INET (IPv4) in struct sockaddr_in.sin_family. + afInet = 2 + // sockaddrInSize is sizeof(struct sockaddr_in) on Linux. + sockaddrInSize = 16 +) + +var ( + // ErrTransparentTCPNotLinux is returned when transparent TCP inbound is + // explicitly enabled on a non-Linux OS. + ErrTransparentTCPNotLinux = errors.New("transparent TCP inbound (SO_ORIGINAL_DST) is only supported on Linux") + // ErrTransparentTCPIPv6Unsupported is returned when an IPv6 bind address or + // IPv6 redirected socket is encountered (P4.1b covers IPv4 only). + ErrTransparentTCPIPv6Unsupported = errors.New("transparent TCP inbound currently supports IPv4 only; IPv6 is not covered") +) + +// TransparentTCPConfig configures the opt-in Linux transparent TCP listener +// fed by iptables/nftables REDIRECT rules. +type TransparentTCPConfig struct { + // Addr is the IPv4 TCP bind address (default DefaultTransparentTCPAddr). + Addr string + // MaxConcurrent bounds simultaneous active transparent streams (default 64). + MaxConcurrent int + // MaxBytesPerDir caps captured payload bytes per direction (default 4 MiB). + MaxBytesPerDir int64 + // PreviewBytes caps inline SQLite preview per direction before blob spill (default 8 KiB). + PreviewBytes int + // IdleTimeout bounds per-direction read/write inactivity during relay (default 2m). + IdleTimeout time.Duration + // HandshakeTimeout bounds original-destination recovery before dialing (default 10s). + HandshakeTimeout time.Duration + // DialTimeout bounds outbound TCP connect attempts (default 30s). + DialTimeout time.Duration + // PaceTimeout bounds waiting on the egress governor before refusing (default 30s). + PaceTimeout time.Duration + // DialContext dials the recovered original destination. In production the + // server wires intercept.Interceptor.DialSOCKS5 so asset block rules and + // target pinning are enforced before any socket is opened. + DialContext func(ctx context.Context, network, addr string) (net.Conn, error) + // BaseContext optionally supplies a parent context for outbound dials. + BaseContext func() context.Context + + // origDst overrides kernel SO_ORIGINAL_DST lookup in unit tests. + origDst func(conn net.Conn) (net.IP, int, error) +} + +// TransparentTCPConfigFromEnv parses ARTEX_TRANSPARENT_TCP_ENABLED and +// ARTEX_TRANSPARENT_TCP_ADDR. Returns (cfg, false) when disabled (default). +func TransparentTCPConfigFromEnv() (TransparentTCPConfig, bool) { + rawEnabled := strings.ToLower(strings.TrimSpace(os.Getenv(TransparentTCPEnabledEnv))) + rawAddr := strings.TrimSpace(os.Getenv(TransparentTCPAddrEnv)) + switch rawEnabled { + case "0", "false", "no", "off": + return TransparentTCPConfig{}, false + case "1", "true", "yes", "on": + addr := rawAddr + if addr == "" { + addr = DefaultTransparentTCPAddr + } + return TransparentTCPConfig{Addr: addr}, true + case "": + if rawAddr == "" { + return TransparentTCPConfig{}, false + } + return TransparentTCPConfig{Addr: rawAddr}, true + default: + log.Printf("[traffic] %s=%q is not a boolean; leaving transparent TCP disabled", TransparentTCPEnabledEnv, rawEnabled) + return TransparentTCPConfig{}, false + } +} + +func (c TransparentTCPConfig) withDefaults() TransparentTCPConfig { + if strings.TrimSpace(c.Addr) == "" { + c.Addr = DefaultTransparentTCPAddr + } + if c.MaxConcurrent <= 0 { + c.MaxConcurrent = DefaultSOCKS5MaxConcurrent + } + if c.MaxBytesPerDir <= 0 { + c.MaxBytesPerDir = DefaultSOCKS5MaxBytesPerDir + } + if c.PreviewBytes <= 0 { + c.PreviewBytes = DefaultSOCKS5PreviewBytes + } + if c.IdleTimeout <= 0 { + c.IdleTimeout = DefaultSOCKS5IdleTimeout + } + if c.HandshakeTimeout <= 0 { + c.HandshakeTimeout = DefaultSOCKS5HandshakeTimeout + } + if c.DialTimeout <= 0 { + c.DialTimeout = DefaultSOCKS5DialTimeout + } + if c.PaceTimeout <= 0 { + c.PaceTimeout = DefaultSOCKS5PaceTimeout + } + if c.DialContext == nil { + var d net.Dialer + c.DialContext = d.DialContext + } + if c.origDst == nil { + c.origDst = originalDstIPv4 + } + return c +} + +// parseOriginalDstIPv4 parses a Linux struct sockaddr_in returned by +// getsockopt(SOL_IP, SO_ORIGINAL_DST) into an IPv4 address and TCP port. +// +// Layout on Linux (16 bytes): +// - bytes 0..1: sin_family (host-endian uint16, must be AF_INET = 2) +// - bytes 2..3: sin_port (network/big-endian uint16, 1..65535) +// - bytes 4..7: sin_addr (4-byte IPv4 address, must not be 0.0.0.0) +// - bytes 8..15: sin_zero +func parseOriginalDstIPv4(raw []byte) (net.IP, int, error) { + if len(raw) < sockaddrInSize { + return nil, 0, fmt.Errorf("short SO_ORIGINAL_DST buffer: got %d bytes, want %d", len(raw), sockaddrInSize) + } + family := binary.NativeEndian.Uint16(raw[0:2]) + if family != afInet { + if family == 10 { // Linux AF_INET6 + return nil, 0, ErrTransparentTCPIPv6Unsupported + } + return nil, 0, fmt.Errorf("unsupported SO_ORIGINAL_DST address family %d (want AF_INET=%d)", family, afInet) + } + port := int(binary.BigEndian.Uint16(raw[2:4])) + if port <= 0 || port > 65535 { + return nil, 0, fmt.Errorf("invalid SO_ORIGINAL_DST port %d", port) + } + ip := net.IPv4(raw[4], raw[5], raw[6], raw[7]).To4() + if ip == nil || ip.IsUnspecified() { + return nil, 0, fmt.Errorf("invalid SO_ORIGINAL_DST IPv4 address %v", ip) + } + return ip, port, nil +} + +// isSelfRedirect reports whether (origIP, origPort) points back at the +// transparent listener itself (which happens when a client connects directly +// to the transparent port without an iptables/nftables REDIRECT rule). +func isSelfRedirect(origIP net.IP, origPort int, localAddr net.Addr) bool { + if origIP == nil || origPort <= 0 || localAddr == nil { + return false + } + localTCP, ok := localAddr.(*net.TCPAddr) + if !ok || localTCP == nil || localTCP.Port != origPort { + return false + } + if localTCP.IP == nil || localTCP.IP.IsUnspecified() { + return true + } + return origIP.Equal(localTCP.IP) +} + +type transparentServer struct { + t *Traffic + cfg TransparentTCPConfig + ln net.Listener + sem chan struct{} + + stopOnce sync.Once + stop chan struct{} + wg sync.WaitGroup + + mu sync.Mutex + active map[net.Conn]struct{} +} + +// StartTransparentTCP binds and starts the opt-in Linux transparent TCP +// inbound listener on t. Returns the bound listener address. On non-Linux OSes +// it returns ErrTransparentTCPNotLinux unless a test origDst hook is supplied. +func (t *Traffic) StartTransparentTCP(cfg TransparentTCPConfig) (string, error) { + if !transparentTCPSupported && cfg.origDst == nil { + return "", ErrTransparentTCPNotLinux + } + cfg = cfg.withDefaults() + host, _, err := net.SplitHostPort(cfg.Addr) + if err != nil { + return "", fmt.Errorf("invalid transparent TCP listen address %q: %w", cfg.Addr, err) + } + if ip := net.ParseIP(strings.Trim(host, "[]")); ip != nil && ip.To4() == nil { + return "", ErrTransparentTCPIPv6Unsupported + } + ln, err := net.Listen("tcp4", cfg.Addr) + if err != nil { + return "", fmt.Errorf("listen transparent TCP on %s: %w", cfg.Addr, err) + } + srv := &transparentServer{ + t: t, + cfg: cfg, + ln: ln, + sem: make(chan struct{}, cfg.MaxConcurrent), + stop: make(chan struct{}), + active: make(map[net.Conn]struct{}), + } + + t.transparentMu.Lock() + if t.transparent != nil { + t.transparentMu.Unlock() + _ = ln.Close() + return "", fmt.Errorf("transparent TCP listener already running on %s", t.transparent.ln.Addr().String()) + } + t.transparent = srv + t.transparentMu.Unlock() + + srv.wg.Add(1) + go srv.serve() + return ln.Addr().String(), nil +} + +// TransparentTCPAddr returns the bound transparent TCP listener address, or "" +// when disabled. +func (t *Traffic) TransparentTCPAddr() string { + t.transparentMu.Lock() + defer t.transparentMu.Unlock() + if t.transparent == nil || t.transparent.ln == nil { + return "" + } + return t.transparent.ln.Addr().String() +} + +func (s *transparentServer) Close() error { + var err error + s.stopOnce.Do(func() { + close(s.stop) + err = s.ln.Close() + s.mu.Lock() + for c := range s.active { + _ = c.Close() + } + s.mu.Unlock() + }) + s.wg.Wait() + return err +} + +func (s *transparentServer) stopping() bool { + select { + case <-s.stop: + return true + default: + return false + } +} + +func (s *transparentServer) trackConn(c net.Conn) bool { + s.mu.Lock() + defer s.mu.Unlock() + if s.stopping() { + return false + } + s.active[c] = struct{}{} + return true +} + +func (s *transparentServer) untrackConn(c net.Conn) { + s.mu.Lock() + delete(s.active, c) + s.mu.Unlock() +} + +func (s *transparentServer) serve() { + defer s.wg.Done() + for { + conn, err := s.ln.Accept() + if err != nil { + if s.stopping() || errors.Is(err, net.ErrClosed) { + return + } + var ne net.Error + if errors.As(err, &ne) && ne.Timeout() { + continue + } + log.Printf("[traffic/transparent] accept error: %v", err) + return + } + if !s.trackConn(conn) { + _ = conn.Close() + return + } + s.wg.Add(1) + go func(c net.Conn) { + defer s.wg.Done() + defer s.untrackConn(c) + defer c.Close() + s.handleConn(c) + }(conn) + } +} + +func (s *transparentServer) handleConn(client net.Conn) { + // 1. Recover original IPv4 destination under HandshakeTimeout. + _ = client.SetDeadline(time.Now().Add(s.cfg.HandshakeTimeout)) + + origIP, origPort, err := s.recoverOriginalDst(client) + if err != nil { + return + } + if isSelfRedirect(origIP, origPort, s.ln.Addr()) { + return + } + + // 2. Enforce max concurrent streams before touching policy or dialing. + select { + case s.sem <- struct{}{}: + defer func() { <-s.sem }() + default: + return + } + + // 3. Apply the same egress governor and target gate before dialing. + host := origIP.String() + targetAddr := net.JoinHostPort(host, strconv.Itoa(origPort)) + upstream, err := s.dialTarget(targetAddr) + if err != nil { + return + } + if !s.trackConn(upstream) { + _ = upstream.Close() + return + } + defer s.untrackConn(upstream) + defer upstream.Close() + + // Clear handshake deadline before starting the relay. + _ = client.SetDeadline(time.Time{}) + + // 4. Bidirectional half-close-aware relay with idle timeout and bounded capture. + startedAt := time.Now().UTC() + c2sCap := &limitedCapture{max: s.cfg.MaxBytesPerDir} + s2cCap := &limitedCapture{max: s.cfg.MaxBytesPerDir} + + var relayWG sync.WaitGroup + relayWG.Add(2) + go func() { + defer relayWG.Done() + relayHalf(upstream, client, c2sCap, s.cfg.IdleTimeout) + closeWrite(upstream) + }() + go func() { + defer relayWG.Done() + relayHalf(client, upstream, s2cCap, s.cfg.IdleTimeout) + closeWrite(client) + }() + relayWG.Wait() + endedAt := time.Now().UTC() + + // 5. Record one raw_tcp_streams row in SQLite (never an HTTP exchanges row). + if s.t != nil { + if _, err := s.t.RecordRawTCPStream(RawTCPCapture{ + Host: host, + Port: origPort, + StartedAt: startedAt, + EndedAt: endedAt, + C2SBytes: c2sCap.total, + S2CBytes: s2cCap.total, + Truncated: c2sCap.truncated || s2cCap.truncated, + C2SData: c2sCap.buf, + S2CData: s2cCap.buf, + }, s.cfg.PreviewBytes); err != nil { + log.Printf("[traffic/transparent] failed to record raw TCP stream %s: %v", targetAddr, err) + } + } +} + +func (s *transparentServer) recoverOriginalDst(client net.Conn) (net.IP, int, error) { + type res struct { + ip net.IP + port int + err error + } + ch := make(chan res, 1) + go func() { + ip, port, err := s.cfg.origDst(client) + ch <- res{ip: ip, port: port, err: err} + }() + timer := time.NewTimer(s.cfg.HandshakeTimeout) + defer timer.Stop() + select { + case r := <-ch: + if r.err != nil { + return nil, 0, r.err + } + if r.ip == nil || r.ip.To4() == nil { + return nil, 0, ErrTransparentTCPIPv6Unsupported + } + if r.port <= 0 || r.port > 65535 { + return nil, 0, fmt.Errorf("invalid original destination port %d", r.port) + } + if r.ip.To4().IsUnspecified() { + return nil, 0, fmt.Errorf("unspecified original destination IP") + } + return r.ip.To4(), r.port, nil + case <-timer.C: + return nil, 0, context.DeadlineExceeded + case <-s.stop: + return nil, 0, net.ErrClosed + } +} + +func (s *transparentServer) dialTarget(targetAddr string) (net.Conn, error) { + baseCtx := context.Background() + if s.cfg.BaseContext != nil { + if ctx := s.cfg.BaseContext(); ctx != nil { + baseCtx = ctx + } + } + stopCtx, stopCancel := context.WithCancel(baseCtx) + defer stopCancel() + go func() { + select { + case <-s.stop: + stopCancel() + case <-stopCtx.Done(): + } + }() + + // Reuse the exact same egress governor (egress.Do) and target gate dialer + // (intercept.PinnedDialer / Interceptor.DialSOCKS5) used by HTTP and SOCKS5. + govCtx, govCancel := context.WithTimeout(stopCtx, s.cfg.PaceTimeout) + defer govCancel() + + targetURL := &url.URL{Scheme: "tcp", Host: targetAddr} + return egress.Do(egress.Current(), govCtx, targetURL, func() (net.Conn, error) { + dialCtx, dialCancel := context.WithTimeout(stopCtx, s.cfg.DialTimeout) + defer dialCancel() + return s.cfg.DialContext(dialCtx, "tcp", targetAddr) + }) +} diff --git a/traffic/transparent_linux.go b/traffic/transparent_linux.go new file mode 100644 index 00000000..e2181c3a --- /dev/null +++ b/traffic/transparent_linux.go @@ -0,0 +1,58 @@ +//go:build linux + +package traffic + +import ( + "fmt" + "net" + "syscall" + "unsafe" +) + +const ( + transparentTCPSupported = true + // soOriginalDst is Linux SO_ORIGINAL_DST from . + soOriginalDst = 80 +) + +// originalDstIPv4 recovers the pre-REDIRECT IPv4 destination of an accepted +// Linux TCP connection using getsockopt(SOL_IP, SO_ORIGINAL_DST). +func originalDstIPv4(conn net.Conn) (net.IP, int, error) { + if ta, ok := conn.LocalAddr().(*net.TCPAddr); ok && ta.IP != nil && ta.IP.To4() == nil { + return nil, 0, ErrTransparentTCPIPv6Unsupported + } + sc, ok := conn.(syscall.Conn) + if !ok { + return nil, 0, fmt.Errorf("connection %T does not expose syscall.Conn", conn) + } + rc, err := sc.SyscallConn() + if err != nil { + return nil, 0, fmt.Errorf("syscall conn: %w", err) + } + var raw [sockaddrInSize]byte + optlen := uint32(len(raw)) + var sockErr error + if err := rc.Control(func(fd uintptr) { + _, _, errno := syscall.Syscall6( + syscall.SYS_GETSOCKOPT, + fd, + uintptr(syscall.SOL_IP), + uintptr(soOriginalDst), + uintptr(unsafe.Pointer(&raw[0])), + uintptr(unsafe.Pointer(&optlen)), + 0, + ) + if errno != 0 { + sockErr = errno + } + }); err != nil { + return nil, 0, fmt.Errorf("control fd: %w", err) + } + if sockErr != nil { + return nil, 0, fmt.Errorf("getsockopt(SO_ORIGINAL_DST): %w", sockErr) + } + if int(optlen) > len(raw) { + optlen = uint32(len(raw)) + } + return parseOriginalDstIPv4(raw[:optlen]) +} diff --git a/traffic/transparent_linux_test.go b/traffic/transparent_linux_test.go new file mode 100644 index 00000000..9cf2348e --- /dev/null +++ b/traffic/transparent_linux_test.go @@ -0,0 +1,167 @@ +//go:build linux + +package traffic + +import ( + "bytes" + "fmt" + "io" + "net" + "os" + "os/exec" + "strconv" + "testing" + "time" + + "github.com/Autumn-27/artex/egress" +) + +const transparentRedirectChildEnv = "ARTEX_TRANSPARENT_REDIRECT_CHILD" + +// TestTransparentTCPLinuxRealIptablesRedirect exercises real Linux kernel +// iptables REDIRECT + getsockopt(SOL_IP, SO_ORIGINAL_DST) inside an isolated +// user+network namespace (unshare -rn) so the host firewall is never touched. +// Skips cleanly when unshare/ip/iptables are unavailable in the environment. +func TestTransparentTCPLinuxRealIptablesRedirect(t *testing.T) { + if os.Getenv(transparentRedirectChildEnv) != "1" { + for _, bin := range []string{"unshare", "ip", "iptables"} { + if _, err := exec.LookPath(bin); err != nil { + t.Skipf("skipping real iptables REDIRECT test: %s not found: %v", bin, err) + } + } + probe := exec.Command("unshare", "-rn", "sh", "-c", "ip link set lo up && iptables -t nat -L OUTPUT -n >/dev/null") + if out, err := probe.CombinedOutput(); err != nil { + t.Skipf("skipping real iptables REDIRECT test: isolated netns/iptables unavailable: %v (%s)", err, bytes.TrimSpace(out)) + } + + cmd := exec.Command("unshare", "-rn", os.Args[0], "-test.v", "-test.run", "^TestTransparentTCPLinuxRealIptablesRedirect$") + cmd.Env = append(os.Environ(), transparentRedirectChildEnv+"=1") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("isolated netns iptables REDIRECT test failed: %v\n%s", err, string(out)) + } + return + } + + // Inside isolated user+network namespace: bring up loopback. + if out, err := exec.Command("ip", "link", "set", "lo", "up").CombinedOutput(); err != nil { + t.Fatalf("ip link set lo up: %v (%s)", err, string(out)) + } + + prev := egress.CurrentProfile() + t.Cleanup(func() { egress.SetProfile(prev) }) + egress.SetProfile(egress.Profile{Name: "off"}) + + // Upstream target echo server listening on 127.0.0.2:. + echoLn, err := net.Listen("tcp4", "127.0.0.2:0") + if err != nil { + t.Fatalf("listen echo on 127.0.0.2:0: %v", err) + } + defer echoLn.Close() + go func() { + for { + c, err := echoLn.Accept() + if err != nil { + return + } + go func(conn net.Conn) { + defer conn.Close() + req, _ := io.ReadAll(conn) + _, _ = conn.Write(append([]byte("NETNS-ECHO:"), req...)) + }(c) + } + }() + echoPort := echoLn.Addr().(*net.TCPAddr).Port + + // Start transparent TCP listener on 127.0.0.1:0 with NO origDst hook (uses real SO_ORIGINAL_DST). + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + + bound, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "127.0.0.1:0", + }) + if err != nil { + t.Fatalf("StartTransparentTCP: %v", err) + } + transPort := func() int { + _, p, err := net.SplitHostPort(bound) + if err != nil { + t.Fatal(err) + } + n, err := strconv.Atoi(p) + if err != nil { + t.Fatal(err) + } + return n + }() + + // Redirect ONLY packets sourced from 127.0.0.3 destined to 127.0.0.2: + // into the transparent listener on . ARTEX's own outbound dial uses + // default loopback source 127.0.0.1 and therefore bypasses REDIRECT cleanly. + iptArgs := []string{ + "-t", "nat", "-A", "OUTPUT", + "-p", "tcp", + "-s", "127.0.0.3", + "-d", "127.0.0.2", + "--dport", strconv.Itoa(echoPort), + "-j", "REDIRECT", "--to-ports", strconv.Itoa(transPort), + } + if out, err := exec.Command("iptables", iptArgs...).CombinedOutput(); err != nil { + t.Fatalf("iptables %v: %v (%s)", iptArgs, err, string(out)) + } + + // 1. Real redirected client connection to 127.0.0.2: from 127.0.0.3. + dialer := net.Dialer{ + Timeout: 2 * time.Second, + LocalAddr: &net.TCPAddr{IP: net.IPv4(127, 0, 0, 3)}, + } + client, err := dialer.Dial("tcp4", fmt.Sprintf("127.0.0.2:%d", echoPort)) + if err != nil { + t.Fatalf("dial redirected target: %v", err) + } + msg := []byte("real-so-original-dst-payload") + if _, err := client.Write(msg); err != nil { + t.Fatal(err) + } + if err := client.(*net.TCPConn).CloseWrite(); err != nil { + t.Fatal(err) + } + reply, err := io.ReadAll(client) + if err != nil { + t.Fatal(err) + } + _ = client.Close() + want := append([]byte("NETNS-ECHO:"), msg...) + if !bytes.Equal(reply, want) { + t.Fatalf("redirected reply = %q, want %q", reply, want) + } + + // Verify real SO_ORIGINAL_DST recovered 127.0.0.2: and stored raw_tcp_streams row. + streams := waitForRawStreams(t, tr, 1) + st := streams[0] + if st.Protocol != "tcp" || st.Host != "127.0.0.2" || st.Port != echoPort { + t.Fatalf("recorded stream = %+v, want host=127.0.0.2 port=%d", st, echoPort) + } + if st.C2SBytes != int64(len(msg)) || st.S2CBytes != int64(len(want)) { + t.Fatalf("recorded bytes = (%d, %d), want (%d, %d)", st.C2SBytes, st.S2CBytes, len(msg), len(want)) + } + var httpCount int + if err := tr.DB().QueryRow(`SELECT COUNT(*) FROM exchanges`).Scan(&httpCount); err != nil || httpCount != 0 { + t.Fatalf("HTTP exchanges = (%d, %v), want 0", httpCount, err) + } + + // 2. Direct non-redirected connection to the transparent port must be refused as a self-loop. + direct, err := net.DialTimeout("tcp4", bound, 2*time.Second) + if err != nil { + t.Fatalf("dial direct: %v", err) + } + _ = direct.SetReadDeadline(time.Now().Add(2 * time.Second)) + buf := make([]byte, 1) + if _, rerr := direct.Read(buf); rerr == nil { + t.Fatal("expected direct non-redirected connection to transparent port to be refused") + } + _ = direct.Close() +} diff --git a/traffic/transparent_other.go b/traffic/transparent_other.go new file mode 100644 index 00000000..da973488 --- /dev/null +++ b/traffic/transparent_other.go @@ -0,0 +1,11 @@ +//go:build !linux + +package traffic + +import "net" + +const transparentTCPSupported = false + +func originalDstIPv4(_ net.Conn) (net.IP, int, error) { + return nil, 0, ErrTransparentTCPNotLinux +} diff --git a/traffic/transparent_test.go b/traffic/transparent_test.go new file mode 100644 index 00000000..946f0799 --- /dev/null +++ b/traffic/transparent_test.go @@ -0,0 +1,579 @@ +package traffic + +import ( + "bytes" + "context" + "encoding/binary" + "errors" + "io" + "net" + "os" + "path/filepath" + "runtime" + "strconv" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/Autumn-27/artex/egress" + "github.com/Autumn-27/artex/intercept" +) + +func makeSockaddrInBytes(family uint16, port uint16, ip [4]byte) []byte { + var raw [sockaddrInSize]byte + binary.NativeEndian.PutUint16(raw[0:2], family) + binary.BigEndian.PutUint16(raw[2:4], port) + copy(raw[4:8], ip[:]) + return raw[:] +} + +func TestParseOriginalDstIPv4(t *testing.T) { + t.Run("valid_ipv4", func(t *testing.T) { + raw := makeSockaddrInBytes(afInet, 8443, [4]byte{198, 51, 100, 42}) + ip, port, err := parseOriginalDstIPv4(raw) + if err != nil { + t.Fatalf("parseOriginalDstIPv4: %v", err) + } + if ip.String() != "198.51.100.42" || port != 8443 { + t.Fatalf("got (%s, %d), want (198.51.100.42, 8443)", ip, port) + } + }) + + t.Run("valid_port_boundaries", func(t *testing.T) { + for _, wantPort := range []uint16{1, 80, 443, 65535} { + raw := makeSockaddrInBytes(afInet, wantPort, [4]byte{127, 0, 0, 2}) + ip, port, err := parseOriginalDstIPv4(raw) + if err != nil || ip.String() != "127.0.0.2" || port != int(wantPort) { + t.Fatalf("port %d: got (%v, %d, %v)", wantPort, ip, port, err) + } + } + }) + + t.Run("short_buffer", func(t *testing.T) { + for _, n := range []int{0, 2, 8, 15} { + if _, _, err := parseOriginalDstIPv4(make([]byte, n)); err == nil { + t.Fatalf("expected error for %d-byte buffer", n) + } + } + }) + + t.Run("ipv6_family_rejected", func(t *testing.T) { + raw := makeSockaddrInBytes(10, 443, [4]byte{198, 51, 100, 1}) + _, _, err := parseOriginalDstIPv4(raw) + if !errors.Is(err, ErrTransparentTCPIPv6Unsupported) { + t.Fatalf("err = %v, want ErrTransparentTCPIPv6Unsupported", err) + } + }) + + t.Run("unknown_family_rejected", func(t *testing.T) { + raw := makeSockaddrInBytes(99, 443, [4]byte{198, 51, 100, 1}) + if _, _, err := parseOriginalDstIPv4(raw); err == nil { + t.Fatal("expected error for unknown address family") + } + }) + + t.Run("port_zero_rejected", func(t *testing.T) { + raw := makeSockaddrInBytes(afInet, 0, [4]byte{198, 51, 100, 1}) + if _, _, err := parseOriginalDstIPv4(raw); err == nil { + t.Fatal("expected error for port 0") + } + }) + + t.Run("unspecified_ip_rejected", func(t *testing.T) { + raw := makeSockaddrInBytes(afInet, 80, [4]byte{0, 0, 0, 0}) + if _, _, err := parseOriginalDstIPv4(raw); err == nil { + t.Fatal("expected error for 0.0.0.0") + } + }) +} + +func TestIsSelfRedirect(t *testing.T) { + localLoopback := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 1081} + localAny := &net.TCPAddr{IP: net.IPv4zero, Port: 1081} + + cases := []struct { + name string + ip net.IP + port int + local net.Addr + want bool + }{ + {"exact_self_loop", net.IPv4(127, 0, 0, 1), 1081, localLoopback, true}, + {"wildcard_same_port", net.IPv4(198, 51, 100, 9), 1081, localAny, true}, + {"different_port", net.IPv4(127, 0, 0, 1), 8080, localLoopback, false}, + {"different_loopback_ip_same_port", net.IPv4(127, 0, 0, 2), 1081, localLoopback, false}, + {"wildcard_different_port", net.IPv4(198, 51, 100, 9), 443, localAny, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := isSelfRedirect(tc.ip, tc.port, tc.local); got != tc.want { + t.Fatalf("isSelfRedirect(%s:%d, %v) = %v, want %v", tc.ip, tc.port, tc.local, got, tc.want) + } + }) + } +} + +func TestTransparentTCPFlagOffAndIPv6BindRejected(t *testing.T) { + t.Setenv(TransparentTCPEnabledEnv, "") + t.Setenv(TransparentTCPAddrEnv, "") + if cfg, ok := TransparentTCPConfigFromEnv(); ok || cfg.Addr != "" { + t.Fatalf("default TransparentTCPConfigFromEnv = (%+v, %v), want disabled", cfg, ok) + } + + t.Setenv(TransparentTCPEnabledEnv, "0") + t.Setenv(TransparentTCPAddrEnv, "127.0.0.1:1081") + if _, ok := TransparentTCPConfigFromEnv(); ok { + t.Fatal("explicit ARTEX_TRANSPARENT_TCP_ENABLED=0 must stay disabled even when addr is set") + } + + t.Setenv(TransparentTCPEnabledEnv, "1") + t.Setenv(TransparentTCPAddrEnv, "") + cfg, ok := TransparentTCPConfigFromEnv() + if !ok || cfg.Addr != DefaultTransparentTCPAddr { + t.Fatalf("enabled default = (%+v, %v), want (%s, true)", cfg, ok, DefaultTransparentTCPAddr) + } + + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + if got := tr.TransparentTCPAddr(); got != "" { + t.Fatalf("TransparentTCPAddr with flag off = %q, want empty", got) + } + + // IPv6 bind address must be rejected explicitly (P4.1b covers IPv4 only). + if _, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "[::1]:0", + origDst: func(net.Conn) (net.IP, int, error) { return net.IPv4(127, 0, 0, 1), 80, nil }, + }); !errors.Is(err, ErrTransparentTCPIPv6Unsupported) { + t.Fatalf("IPv6 bind err = %v, want ErrTransparentTCPIPv6Unsupported", err) + } +} + +func TestTransparentTCPRelayHalfCloseAndStoreRow(t *testing.T) { + prev := egress.CurrentProfile() + t.Cleanup(func() { egress.SetProfile(prev) }) + egress.SetProfile(egress.Profile{Name: "off"}) + + // Echo-after-EOF server to verify bidirectional half-close propagation. + ln, err := net.Listen("tcp4", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + go func() { + for { + c, err := ln.Accept() + if err != nil { + return + } + go func(conn net.Conn) { + defer conn.Close() + req, _ := io.ReadAll(conn) + _, _ = conn.Write(append([]byte("TPROXY-REPLY:"), req...)) + }(c) + } + }() + echoAddr := ln.Addr().(*net.TCPAddr) + + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + + listenAddr, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "127.0.0.1:0", + origDst: func(net.Conn) (net.IP, int, error) { + return echoAddr.IP.To4(), echoAddr.Port, nil + }, + }) + if err != nil { + t.Fatal(err) + } + + c, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + defer c.Close() + + payload := []byte("hello-transparent-tcp") + if _, err := c.Write(payload); err != nil { + t.Fatal(err) + } + if err := c.(*net.TCPConn).CloseWrite(); err != nil { + t.Fatal(err) + } + got, err := io.ReadAll(c) + if err != nil { + t.Fatal(err) + } + want := append([]byte("TPROXY-REPLY:"), payload...) + if !bytes.Equal(got, want) { + t.Fatalf("reply = %q, want %q", got, want) + } + _ = c.Close() + + streams := waitForRawStreams(t, tr, 1) + s := streams[0] + if s.Protocol != "tcp" || s.Host != "127.0.0.1" || s.Port != echoAddr.Port { + t.Fatalf("unexpected stream metadata: %+v", s) + } + if s.C2SBytes != int64(len(payload)) || s.S2CBytes != int64(len(want)) || s.Truncated { + t.Fatalf("unexpected stream counters: %+v", s) + } + if !bytes.Equal(s.C2SPreview, payload) || !bytes.Equal(s.S2CPreview, want) { + t.Fatalf("preview mismatch: c2s=%q s2c=%q", s.C2SPreview, s.S2CPreview) + } + + // Raw TCP stream must NEVER create an HTTP exchanges row. + var httpRows int + if err := tr.DB().QueryRow(`SELECT COUNT(*) FROM exchanges`).Scan(&httpRows); err != nil { + t.Fatal(err) + } + if httpRows != 0 { + t.Fatalf("HTTP exchanges rows = %d, want 0", httpRows) + } +} + +func TestTransparentTCPMalformedOrigDstAndSelfLoopRefusedBeforeDial(t *testing.T) { + prev := egress.CurrentProfile() + t.Cleanup(func() { egress.SetProfile(prev) }) + egress.SetProfile(egress.Profile{Name: "off"}) + + cases := []struct { + name string + origDst func(listenAddr *net.TCPAddr) func(net.Conn) (net.IP, int, error) + }{ + { + name: "getsockopt_error", + origDst: func(*net.TCPAddr) func(net.Conn) (net.IP, int, error) { + return func(net.Conn) (net.IP, int, error) { return nil, 0, errors.New("ENOENT") } + }, + }, + { + name: "ipv6_orig_dst_rejected", + origDst: func(*net.TCPAddr) func(net.Conn) (net.IP, int, error) { + return func(net.Conn) (net.IP, int, error) { return net.ParseIP("2001:db8::1"), 443, nil } + }, + }, + { + name: "port_zero_rejected", + origDst: func(*net.TCPAddr) func(net.Conn) (net.IP, int, error) { + return func(net.Conn) (net.IP, int, error) { return net.IPv4(198, 51, 100, 1), 0, nil } + }, + }, + { + name: "unspecified_ip_rejected", + origDst: func(*net.TCPAddr) func(net.Conn) (net.IP, int, error) { + return func(net.Conn) (net.IP, int, error) { return net.IPv4zero, 80, nil } + }, + }, + { + name: "self_loop_rejected", + origDst: func(lnAddr *net.TCPAddr) func(net.Conn) (net.IP, int, error) { + return func(net.Conn) (net.IP, int, error) { return lnAddr.IP.To4(), lnAddr.Port, nil } + }, + }, + } + + baseGoroutines := runtime.NumGoroutine() + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + + var dialCalls atomic.Int32 + var boundTCP atomic.Pointer[net.TCPAddr] + listenAddr, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "127.0.0.1:0", + HandshakeTimeout: 500 * time.Millisecond, + DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + dialCalls.Add(1) + return nil, errors.New("must not dial") + }, + origDst: func(c net.Conn) (net.IP, int, error) { + return tc.origDst(boundTCP.Load())(c) + }, + }) + if err != nil { + t.Fatal(err) + } + la, err := net.ResolveTCPAddr("tcp4", listenAddr) + if err != nil { + t.Fatal(err) + } + boundTCP.Store(la) + + c, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + _ = c.SetReadDeadline(time.Now().Add(2 * time.Second)) + buf := make([]byte, 1) + if _, rerr := c.Read(buf); rerr == nil { + t.Fatal("expected connection to be closed on malformed/self-loop origDst") + } + _ = c.Close() + + if dialCalls.Load() != 0 { + t.Fatalf("DialContext called %d times, want 0", dialCalls.Load()) + } + n, err := tr.CountRawTCPStreams("") + if err != nil || n != 0 { + t.Fatalf("CountRawTCPStreams = (%d, %v), want 0", n, err) + } + }) + } + + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if runtime.NumGoroutine() <= baseGoroutines+10 { + return + } + time.Sleep(25 * time.Millisecond) + } + if after := runtime.NumGoroutine(); after > baseGoroutines+10 { + t.Fatalf("possible goroutine leak: before=%d after=%d", baseGoroutines, after) + } +} + +func TestTransparentTCPDeniedTargetAndGovernorRefusedBeforeDial(t *testing.T) { + t.Run("target_gate_refuses_before_dial", func(t *testing.T) { + prev := egress.CurrentProfile() + t.Cleanup(func() { egress.SetProfile(prev) }) + egress.SetProfile(egress.Profile{Name: "off"}) + + echo, accepted := startEchoServer(t, "tcp4", "127.0.0.1:0") + echoAddr := echo.Addr().(*net.TCPAddr) + + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + + // Use intercept.PinnedDialer with a gated TargetScope whose Interceptor is nil: + // PinnedDialer refuses BEFORE dialing any socket. + listenAddr, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "127.0.0.1:0", + BaseContext: func() context.Context { + return intercept.WithTargetScope(context.Background(), intercept.TargetScope{ + TaskID: "99", + Enabled: func() bool { return true }, + }) + }, + DialContext: intercept.PinnedDialer("transparent", true), + origDst: func(net.Conn) (net.IP, int, error) { + return echoAddr.IP.To4(), echoAddr.Port, nil + }, + }) + if err != nil { + t.Fatal(err) + } + + c, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + _ = c.SetReadDeadline(time.Now().Add(2 * time.Second)) + buf := make([]byte, 1) + if _, rerr := c.Read(buf); rerr == nil { + t.Fatal("expected blocked target connection to close immediately") + } + _ = c.Close() + + if accepted.Load() != 0 { + t.Fatalf("target accepted %d connections despite target gate refusal, want 0", accepted.Load()) + } + n, err := tr.CountRawTCPStreams("") + if err != nil || n != 0 { + t.Fatalf("CountRawTCPStreams = (%d, %v), want 0", n, err) + } + }) + + t.Run("governor_refuses_before_dial", func(t *testing.T) { + prev := egress.CurrentProfile() + t.Cleanup(func() { egress.SetProfile(prev) }) + // 1 token burst, very slow refill: 2nd connection exceeds PaceTimeout and is refused BEFORE dial. + egress.SetProfile(egress.Profile{Name: "tight", RPS: 0.01, Burst: 1, MaxPerHost: 1, MaxGlobal: 1}) + + echo, _ := startEchoServer(t, "tcp4", "127.0.0.1:0") + echoAddr := echo.Addr().(*net.TCPAddr) + + var dialCalls atomic.Int32 + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + + listenAddr, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "127.0.0.1:0", + PaceTimeout: 40 * time.Millisecond, + DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + dialCalls.Add(1) + return (&net.Dialer{Timeout: 2 * time.Second}).DialContext(ctx, network, addr) + }, + origDst: func(net.Conn) (net.IP, int, error) { + return echoAddr.IP.To4(), echoAddr.Port, nil + }, + }) + if err != nil { + t.Fatal(err) + } + + // 1st connection consumes the single burst token. + c1, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + _, _ = c1.Write([]byte("ok")) + buf2 := make([]byte, 2) + if _, err := io.ReadFull(c1, buf2); err != nil { + t.Fatalf("first connection failed: %v", err) + } + _ = c1.Close() + + // 2nd connection exceeds PaceTimeout inside egress.Do and is refused BEFORE dial. + before := dialCalls.Load() + c2, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + _ = c2.SetReadDeadline(time.Now().Add(2 * time.Second)) + one := make([]byte, 1) + if _, rerr := c2.Read(one); rerr == nil { + t.Fatal("expected governor-refused connection to close") + } + _ = c2.Close() + + if dialCalls.Load() != before { + t.Fatalf("DialContext called despite governor refusal (before=%d after=%d)", before, dialCalls.Load()) + } + }) +} + +func TestTransparentTCPByteCapsTruncationBlobSpillAndConcurrency(t *testing.T) { + prev := egress.CurrentProfile() + t.Cleanup(func() { egress.SetProfile(prev) }) + egress.SetProfile(egress.Profile{Name: "off"}) + + echo, _ := startEchoServer(t, "tcp4", "127.0.0.1:0") + echoAddr := echo.Addr().(*net.TCPAddr) + + tr, err := Open(t.TempDir(), "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer tr.Close() + + listenAddr, err := tr.StartTransparentTCP(TransparentTCPConfig{ + Addr: "127.0.0.1:0", + MaxConcurrent: 2, + MaxBytesPerDir: 512, + PreviewBytes: 64, + IdleTimeout: 200 * time.Millisecond, + origDst: func(net.Conn) (net.IP, int, error) { + return echoAddr.IP.To4(), echoAddr.Port, nil + }, + }) + if err != nil { + t.Fatal(err) + } + + // 1. Oversized stream -> truncated=true, preview=64, blob spilled atomically with blob_refs. + c, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + big := bytes.Repeat([]byte("TP"), 512) // 1024 bytes > MaxBytesPerDir(512) > PreviewBytes(64) + if _, err := c.Write(big); err != nil { + t.Fatal(err) + } + if err := c.(*net.TCPConn).CloseWrite(); err != nil { + t.Fatal(err) + } + got, err := io.ReadAll(c) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, big) { + t.Fatalf("relayed %d bytes, want %d", len(got), len(big)) + } + _ = c.Close() + + streams := waitForRawStreams(t, tr, 1) + st := streams[0] + if !st.Truncated || st.C2SBytes != 1024 || st.S2CBytes != 1024 { + t.Fatalf("unexpected truncation/counts: %+v", st) + } + if len(st.C2SPreview) != 64 || st.C2SBlob == "" || st.S2CBlob == "" { + t.Fatalf("expected 64-byte preview and blob hashes: %+v", st) + } + blobFile := filepath.Join(tr.dir, "_blobs", "sha256", st.C2SBlob[:2], st.C2SBlob+".bin") + blobBytes, err := os.ReadFile(blobFile) + if err != nil || len(blobBytes) != 512 { + t.Fatalf("blob file (%s): len=%d err=%v, want 512", blobFile, len(blobBytes), err) + } + var refs int + if err := tr.DB().QueryRow(`SELECT COUNT(*) FROM blob_refs WHERE exchange_id = ?`, st.ID).Scan(&refs); err != nil || refs < 1 { + t.Fatalf("blob_refs for %s = (%d, %v), want >= 1", st.ID, refs, err) + } + + // 2. MaxConcurrent=2 + IdleTimeout: hold 2 active streams, verify 3rd is rejected cleanly. + c1, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + defer c1.Close() + c2, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + defer c2.Close() + + // Send one byte on each so both goroutines acquire semaphore slots. + var wg sync.WaitGroup + wg.Add(2) + for _, conn := range []net.Conn{c1, c2} { + go func(cn net.Conn) { + defer wg.Done() + _, _ = cn.Write([]byte("hi")) + buf := make([]byte, 2) + _, _ = io.ReadFull(cn, buf) + }(conn) + } + wg.Wait() + + c3, err := net.DialTimeout("tcp4", listenAddr, 2*time.Second) + if err != nil { + t.Fatal(err) + } + _ = c3.SetReadDeadline(time.Now().Add(2 * time.Second)) + one := make([]byte, 1) + if _, rerr := c3.Read(one); rerr == nil { + t.Fatal("expected 3rd concurrent stream beyond MaxConcurrent=2 to close") + } + _ = c3.Close() + + // IdleTimeout closes c1 and c2 cleanly. + _ = c1.SetReadDeadline(time.Now().Add(2 * time.Second)) + if _, rerr := c1.Read(one); rerr == nil { + t.Fatal("expected idle stream to close after IdleTimeout") + } + _ = c1.Close() + _ = c2.Close() + + all := waitForRawStreams(t, tr, 3) + if len(all) != 3 { + t.Fatalf("want 3 recorded streams (1 big + 2 concurrent), got %d", len(all)) + } + _ = strconv.Itoa(len(all)) +}