Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ All notable changes to this project are documented in this file. The format is b
- **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197).
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
- **Opt-in Linux transparent TCP inbound & raw TCP recording (P4.1b)**: on Linux when traffic capture is enabled, setting `ARTEX_TRANSPARENT_TCP_ENABLED=1` (loopback `127.0.0.1:1081` by default) or `ARTEX_TRANSPARENT_TCP_ADDR=<ipv4:port>` starts an IPv4 transparent TCP listener fed by operator-managed `iptables`/`nftables` `REDIRECT` rules (off by default; ARTEX never touches the host firewall; root/`CAP_NET_ADMIN` is required only for the operator's redirect rules; IPv6 is not covered). Recovers original IPv4 destinations via `SO_ORIGINAL_DST` with self-redirect loop protection, reuses the P4.1a target gate (`intercept.Interceptor.DialSOCKS5`), egress governor (`egress.Do`), half-close relay bounds, and SQLite `raw_tcp_streams` store (atomic `_blobs` spill, no HTTP `exchanges` row, no PostgreSQL migration). See `docs/traffic-transparent.md`.
- **Opt-in SOCKS5 inbound proxy & raw TCP recording (P4.1a)**: when traffic capture is enabled, setting `ARTEX_SOCKS5_ENABLED=1` (loopback `127.0.0.1:1080` by default) or `ARTEX_SOCKS5_ADDR=<addr>` starts an RFC 1928 SOCKS5 `CONNECT` listener alongside the HTTP recording proxy (off by default; HTTP proxy and agent proxy env vars are unchanged; listener does not start if traffic recording is disabled). Accepts loopback no-auth and RFC 1929 username/password without verifying credentials (carrying `artex-pace` in the username; binding to a non-loopback address therefore exposes an unauthenticated proxy, though the target gate and governor still apply), rejects `BIND`/`UDP ASSOCIATE` with `0x07`, enforces the existing target gate (`intercept.Interceptor.DialSOCKS5` / `intercept.PinnedDialer`) and egress governor (`egress.Do`) before dialing, relays half-close-aware TCP with concurrency/byte/idle/handshake bounds, and records one row per connection in SQLite `raw_tcp_streams` with capped directional previews and atomic (temp+rename) `_blobs` refs without writing HTTP `exchanges` rows. See `docs/traffic-socks5.md`.
- **Opt-in worker cache-prefix split (P3.5 Phase C)**: setting `ARTEX_CACHE_PREFIX=1` (off by default) splits the `worker` system prompt at `DynamicBoundary = 1` into a sibling-shareable static prefix (`body + untrustedContentTail + workerTrafficBlock`) and a per-intent dynamic suffix (`workerArtifactSpec(runDir)` through `<available-deferred-tools>`) without reordering sections or altering the visible joined prompt text. Planner, mainagent, chat, and goals are unchanged; custom worker prompts referencing `{{.Now}}` remain volatile in the static segment. See `docs/context-telemetry.md`.
- **Opt-in LLM context size telemetry (P3.5 Phase A)**: setting `ARTEX_CONTEXT_TELEMETRY=1` (off by default) records per-call request section byte sizes (`system_static`/`system_dynamic` split at `DynamicBoundary`, serialized `tools`, `messages` by role, `tool_result` subset, and `last_message_bytes`), tool/message counts, and token/cache usage in the new `llm_context` table (migration `0009_llm_context.sql`), with zero prompt/behavior change and zero content stored, independent of `llm_record`. `GET /api/llm/context/summary` aggregates input percentiles (`p50`/`p90`/`p99`), pro-rata section shares over real `input_tokens`, `cache_read/input` ratios by `(worker, model)`, and top tasks. See `docs/context-telemetry.md`.
Expand Down
81 changes: 81 additions & 0 deletions docs/traffic-transparent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Opt-in Linux Transparent TCP Inbound & Raw TCP Recording (P4.1b)

ARTEX ships an opt-in **Linux-only IPv4 transparent TCP inbound listener** in `traffic/` that accepts connections redirected by `iptables` or `nftables` `REDIRECT` rules, recovers each socket's original IPv4 destination via `getsockopt(SOL_IP, SO_ORIGINAL_DST)`, enforces the existing target gate and egress governor before dialing, and records the connection in SQLite `raw_tcp_streams`.

## Opt-in Configuration (Off by Default)

| Environment Variable | Default | Behavior |
|---|---|---|
| `ARTEX_TRANSPARENT_TCP_ENABLED` | unset (off) | Set to `1` / `true` / `on` / `yes` to start the transparent TCP listener on `127.0.0.1:1081` (IPv4 loopback). Explicit `0` / `false` / `off` / `no` forces it off even if `ARTEX_TRANSPARENT_TCP_ADDR` is set. |
| `ARTEX_TRANSPARENT_TCP_ADDR` | `127.0.0.1:1081` | Explicit IPv4 TCP bind address; setting a non-empty value also enables the listener unless `ARTEX_TRANSPARENT_TCP_ENABLED` is explicitly false. Use `0.0.0.0:1081` only when redirecting traffic from other interfaces (`PREROUTING`) and protect the port with a host firewall. |

- **Flag off**: no transparent TCP listener is started, no firewall rules are touched, and the HTTP MITM proxy is unchanged.
- **Requires traffic recorder**: the server only starts the transparent TCP listener when the traffic recorder is enabled (inside the traffic proxy block in `server/manager.go`). If traffic capture is disabled, setting these environment variables does not open a listener.
- **Linux only**: on non-Linux operating systems (`windows`, `darwin`), leaving the flag off is a no-op; explicitly enabling it logs `ErrTransparentTCPNotLinux` and leaves the listener disabled.
- **IPv4 only (IPv6 not covered)**: P4.1b recovers `SO_ORIGINAL_DST` (`SOL_IP`, `AF_INET`) for IPv4 TCP sockets only. IPv6 (`IP6T_SO_ORIGINAL_DST`) is **not covered**; IPv6 bind addresses and IPv6 sockets are refused.
- **Firewall never touched by ARTEX**: ARTEX never installs, modifies, or deletes `iptables`/`nftables` rules. Root or `CAP_NET_ADMIN` is required **only** by the operator when installing and cleaning up the host redirect rules—never to run ARTEX itself.

## Firewall Rules & Cleanup (Operator-Managed)

Choose **either** `iptables` **or** `nftables`. Always exclude the UID running ARTEX (example: `artex`) from local `OUTPUT` redirect rules so ARTEX's own outbound dials do not loop back into the transparent listener.

### Option 1: `iptables` (IPv4 local `OUTPUT` REDIRECT)

Install into a dedicated chain so cleanup is atomic:

```bash
# Requires root or CAP_NET_ADMIN
sudo iptables -t nat -N ARTEX_TRANSPARENT
# Never redirect ARTEX's own outbound dials or loopback traffic
sudo iptables -t nat -A ARTEX_TRANSPARENT -m owner --uid-owner artex -j RETURN
sudo iptables -t nat -A ARTEX_TRANSPARENT -d 127.0.0.0/8 -j RETURN
# Example: redirect outbound TCP to lab target 198.51.100.0/24 into ARTEX on 127.0.0.1:1081
sudo iptables -t nat -A ARTEX_TRANSPARENT -p tcp -d 198.51.100.0/24 -j REDIRECT --to-ports 1081
sudo iptables -t nat -A OUTPUT -p tcp -j ARTEX_TRANSPARENT
```

Cleanup (exact removal):

```bash
sudo iptables -t nat -D OUTPUT -p tcp -j ARTEX_TRANSPARENT
sudo iptables -t nat -F ARTEX_TRANSPARENT
sudo iptables -t nat -X ARTEX_TRANSPARENT
```

*(For traffic arriving on another interface—for example a container or routed VM—bind `ARTEX_TRANSPARENT_TCP_ADDR=0.0.0.0:1081`, attach `ARTEX_TRANSPARENT` to `PREROUTING` instead of `OUTPUT`, and drop direct non-redirected packets to port 1081 at the host firewall.)*

### Option 2: `nftables` (IPv4 local `OUTPUT` REDIRECT)

Install into a dedicated table so cleanup is a single command:

```bash
# Requires root or CAP_NET_ADMIN
sudo nft -f - <<'NFT'
table ip artex_transparent {
chain output {
type nat hook output priority -100; policy accept;
meta skuid "artex" return
ip daddr 127.0.0.0/8 return
ip daddr 198.51.100.0/24 tcp redirect to :1081
}
}
NFT
```

Cleanup (exact removal):

```bash
sudo nft delete table ip artex_transparent
```

## Policy Enforcement, Loop Guard & Storage (Shared with P4.1a)

For every accepted TCP connection:

1. **Original destination recovery**: reads `getsockopt(SOL_IP, SO_ORIGINAL_DST)` under `HandshakeTimeout` (default `10s`), parses `struct sockaddr_in` (rejecting short buffers, non-`AF_INET` families including IPv6, port `0`, and `0.0.0.0`), and rejects **self-redirects** (direct non-redirected connections where `SO_ORIGINAL_DST` equals the listener's own address/port) before dialing.
2. **Egress governor (`egress.Do(egress.Current(), ...)`)**: enforces the active noise profile (`PaceTimeout`, default `30s`) before dialing.
3. **Target gate (`intercept.Interceptor.DialSOCKS5` / `intercept.PinnedDialer`)**: enforces global asset block rules and any enabled `intercept.TargetScope` pins/exemptions before opening any outbound socket. Because the transparent listener only knows the recovered destination IP (never a DNS hostname), hostname-based block or allow rules cannot match—IP and CIDR rules are the ones that apply. Refused connections are closed immediately without dialing and without writing a store row.
4. **Relay & bounds**: reuses the P4.1a full-duplex half-close-aware relay (`CloseWrite`) and defaults (`MaxConcurrent=64`, `MaxBytesPerDir=4 MiB`, `PreviewBytes=8 KiB`, `IdleTimeout=2m`, `HandshakeTimeout=10s`, `DialTimeout=30s`, `PaceTimeout=30s`).
5. **SQLite storage**: records one row per completed connection in `raw_tcp_streams` (`protocol='tcp'`, atomic temp+rename `_blobs/sha256/` spill + `blob_refs`), with **zero HTTP `exchanges` rows** and **no PostgreSQL migration**.

Closing `Traffic` stops accepting new transparent connections, closes active sockets, and waits for all handler goroutines to exit before closing SQLite.
8 changes: 8 additions & 0 deletions server/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,14 @@ func NewManager(dir, proxyAddr string) (*Manager, error) {
log.Printf("[traffic/socks5] opt-in SOCKS5 CONNECT listener on %s", bound)
}
}
if tcfg, ok := traffic.TransparentTCPConfigFromEnv(); ok {
tcfg.DialContext = m.interceptor.DialSOCKS5
if bound, terr := tr.StartTransparentTCP(tcfg); terr != nil {
log.Printf("[traffic/transparent] disabled: %v", terr)
} else {
log.Printf("[traffic/transparent] opt-in transparent TCP listener on %s (Linux IPv4 SO_ORIGINAL_DST)", bound)
}
}
go func() {
log.Printf("[traffic] recording proxy on %s (set HTTP_PROXY=%s + trust _ca CA)", proxyAddr, tr.ProxyAddr())
if err := tr.Start(); err != nil {
Expand Down
48 changes: 48 additions & 0 deletions server/transparent_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package server

import (
"runtime"
"testing"

"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/traffic"
)

func TestManagerTransparentTCPOptInFlag(t *testing.T) {
// 1. Flag off (default): Manager starts NO transparent TCP listener.
t.Setenv(traffic.TransparentTCPEnabledEnv, "")
t.Setenv(traffic.TransparentTCPAddrEnv, "")

mOff, err := NewManager(t.TempDir(), "127.0.0.1:0")
if err != nil {
db.SkipPGUnavailable(t, err)
}
defer mOff.Close()
if mOff.traffic == nil {
t.Fatal("expected traffic recorder to be initialized")
}
if got := mOff.traffic.TransparentTCPAddr(); got != "" {
t.Fatalf("TransparentTCPAddr with flag off = %q, want empty", got)
}

// 2. Flag on (ARTEX_TRANSPARENT_TCP_ADDR=127.0.0.1:0): starts on Linux,
// stays cleanly disabled on non-Linux without crashing NewManager.
t.Setenv(traffic.TransparentTCPAddrEnv, "127.0.0.1:0")

mOn, err := NewManager(t.TempDir(), "127.0.0.1:0")
if err != nil {
db.SkipPGUnavailable(t, err)
}
defer mOn.Close()

gotAddr := mOn.traffic.TransparentTCPAddr()
if runtime.GOOS == "linux" {
if gotAddr == "" {
t.Fatal("expected transparent TCP listener on Linux when ARTEX_TRANSPARENT_TCP_ADDR is set")
}
} else {
if gotAddr != "" {
t.Fatalf("expected transparent TCP listener disabled on %s, got %q", runtime.GOOS, gotAddr)
}
}
}
11 changes: 11 additions & 0 deletions traffic/traffic.go
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,10 @@ type Traffic struct {
// socks holds the opt-in SOCKS5 CONNECT listener when started via StartSOCKS5.
socksMu sync.Mutex
socks *socks5Server
// transparent holds the opt-in Linux transparent TCP listener when started
// via StartTransparentTCP (P4.1b).
transparentMu sync.Mutex
transparent *transparentServer
}

// Open initializes the traffic tree, blob store and SQLite index under dir.
Expand Down Expand Up @@ -478,6 +482,13 @@ func (t *Traffic) Close() error {
if srv != nil {
_ = srv.Close()
}
t.transparentMu.Lock()
tsrv := t.transparent
t.transparent = nil
t.transparentMu.Unlock()
if tsrv != nil {
_ = tsrv.Close()
}
t.reaping.Wait()
t.sweeping.Wait()
return t.db.Close()
Expand Down
Loading
Loading