Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ All notable changes to this project are documented in this file. The format is b
- **`ask_operator` target warning**: a request whose question names a host or URL outside the goal targets is tagged as containing target-derived text (#197).
- **Opt-in update banner**: `ARTEX_UPDATE_CHECK` makes the backend check the latest GitHub release at most once per 24 hours and the UI show a banner; off by default (#202).
- **Release provenance**: release archives carry GitHub build provenance attestations and a CycloneDX SBOM (`gh attestation verify <zip> --repo BaaRai/ARTEX`); release notes are the matching `CHANGELOG.md` section (#219, #201).
- **Opt-in SOCKS5 inbound proxy & raw TCP recording (P4.1a)**: when traffic capture is enabled, setting `ARTEX_SOCKS5_ENABLED=1` (loopback `127.0.0.1:1080` by default) or `ARTEX_SOCKS5_ADDR=<addr>` starts an RFC 1928 SOCKS5 `CONNECT` listener alongside the HTTP recording proxy (off by default; HTTP proxy and agent proxy env vars are unchanged; listener does not start if traffic recording is disabled). Accepts loopback no-auth and RFC 1929 username/password without verifying credentials (carrying `artex-pace` in the username; binding to a non-loopback address therefore exposes an unauthenticated proxy, though the target gate and governor still apply), rejects `BIND`/`UDP ASSOCIATE` with `0x07`, enforces the existing target gate (`intercept.Interceptor.DialSOCKS5` / `intercept.PinnedDialer`) and egress governor (`egress.Do`) before dialing, relays half-close-aware TCP with concurrency/byte/idle/handshake bounds, and records one row per connection in SQLite `raw_tcp_streams` with capped directional previews and atomic (temp+rename) `_blobs` refs without writing HTTP `exchanges` rows. See `docs/traffic-socks5.md`.
- **Opt-in worker cache-prefix split (P3.5 Phase C)**: setting `ARTEX_CACHE_PREFIX=1` (off by default) splits the `worker` system prompt at `DynamicBoundary = 1` into a sibling-shareable static prefix (`body + untrustedContentTail + workerTrafficBlock`) and a per-intent dynamic suffix (`workerArtifactSpec(runDir)` through `<available-deferred-tools>`) without reordering sections or altering the visible joined prompt text. Planner, mainagent, chat, and goals are unchanged; custom worker prompts referencing `{{.Now}}` remain volatile in the static segment. See `docs/context-telemetry.md`.
- **Opt-in LLM context size telemetry (P3.5 Phase A)**: setting `ARTEX_CONTEXT_TELEMETRY=1` (off by default) records per-call request section byte sizes (`system_static`/`system_dynamic` split at `DynamicBoundary`, serialized `tools`, `messages` by role, `tool_result` subset, and `last_message_bytes`), tool/message counts, and token/cache usage in the new `llm_context` table (migration `0009_llm_context.sql`), with zero prompt/behavior change and zero content stored, independent of `llm_record`. `GET /api/llm/context/summary` aggregates input percentiles (`p50`/`p90`/`p99`), pro-rata section shares over real `input_tokens`, `cache_read/input` ratios by `(worker, model)`, and top tasks. See `docs/context-telemetry.md`.
- **HAR 1.2 traffic import and export**: `POST /api/traffic/import/har` imports raw HAR 1.2 JSON captures (up to 32 MiB / 2,000 entries / 32 MiB decoded bodies) with deterministic `har-<sha256>` IDs (and `_artexExchangeId` preservation for ARTEX exports so distinct same-second exchanges round-trip without collapsing), base64 decoding, `postData.params` and `cookies` array fallbacks, URL userinfo stripping, blob spilling above 256 KiB, and full-text indexing, without applying capture rewrite rules (`exchange_rewrites` stays empty); per-entry errors are reported alongside `imported`/`skipped`/`failed` counts. `GET /api/traffic/export/har` exports filtered traffic (reusing the `GET /api/traffic` filters) as a compliant HAR 1.2 document (with explicit unknown placeholders for unrecorded timings and header sizes) in stable `(ts, id)` order, encoding UTF-8 bodies as text and binary bodies as base64, capped at 1,000 entries / 20 MiB raw headers+bodies (31 MiB JSON backstop, guaranteed under the 32 MiB import limit) with explicit truncation reporting via `X-ARTEX-HAR-*` headers and `_truncated`/`_truncatedReason`/`_totalMatched` fields. The Traffic page adds **Import HAR** and **Export HAR** controls. See `docs/traffic-har.md`.
Expand Down
56 changes: 56 additions & 0 deletions docs/traffic-socks5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Opt-in SOCKS5 Inbound Proxy & Raw TCP Recording (P4.1a)

ARTEX ships an opt-in **RFC 1928 SOCKS5 `CONNECT`** inbound listener in `traffic/` that records raw TCP connections separately from HTTP traffic.

## Opt-in Configuration (Off by Default)

| Environment Variable | Default | Behavior |
|---|---|---|
| `ARTEX_SOCKS5_ENABLED` | unset (off) | Set to `1` / `true` / `on` / `yes` to start the SOCKS5 listener on `127.0.0.1:1080` (loopback). Explicit `0` / `false` / `off` / `no` forces it off even if `ARTEX_SOCKS5_ADDR` is set. |
| `ARTEX_SOCKS5_ADDR` | `127.0.0.1:1080` | Explicit TCP bind address; setting a non-empty value also enables the listener unless `ARTEX_SOCKS5_ENABLED` is explicitly false. |

- **Flag off**: no SOCKS5 listener is started and the HTTP MITM recording proxy is byte-identical.
- **Requires traffic recorder**: the server only starts the SOCKS5 listener when the traffic recorder is enabled (inside the traffic proxy block in `server/manager.go`). If traffic capture is disabled, setting `ARTEX_SOCKS5_ENABLED` / `ARTEX_SOCKS5_ADDR` does not open a listener.
- **Agents unchanged**: workers and tools keep `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY` pointing to the existing HTTP proxy; ARTEX does not silently switch agents to SOCKS5.
- **Non-loopback bind warning**: RFC 1929 username/password (`0x02`) accepts **any** username and password without verifying credentials (so the pace marker `artex-pace` can ride in the username). Binding `ARTEX_SOCKS5_ADDR` to a non-loopback address therefore exposes an **unauthenticated proxy** to anyone who can reach that port. The target gate and egress governor still apply to every connection, but keep the listener on loopback (`127.0.0.1`) or restrict access at the host firewall.

## Protocol Scope (P4.1a)

- **Commands**: `CONNECT` (`0x01`) only. `BIND` (`0x02`), `UDP ASSOCIATE` (`0x03`, planned for P4.1c), and unknown commands return RFC 1928 reply `0x07` (*Command not supported*).
- **Addressing**: IPv4 (`0x01`), domain name (`0x03`), and IPv6 (`0x04`). Unknown address types return `0x08` (*Address type not supported*).
- **Authentication**:
- Loopback peers (`net.IP.IsLoopback()`) may connect with **No Auth** (`0x00`). Non-loopback peers offering only `0x00` are rejected with `0x05 0xFF`.
- **Username/Password** (`0x02`, RFC 1929) is accepted without requiring credentials so clients can pass the pace marker (`artex-pace`, `egress.PaceUser`) in the username.
- **Relay**: Full-duplex, half-close aware (`CloseWrite` on EOF in each direction).

## Policy Enforcement Before Dialing

Before opening any outbound TCP socket, the listener invokes the same controls used by the HTTP path:

1. **Egress governor (`egress.Do(egress.Current(), ...)`)**: enforces the active noise profile (`off` is a zero-delay passthrough). If waiting for a slot/token exceeds `PaceTimeout`, the connection is refused before dialing with SOCKS5 reply `0x02` (*Connection not allowed by ruleset*).
2. **Target gate (`intercept.Interceptor.DialSOCKS5` / `intercept.PinnedDialer`)**: enforces global asset block rules (before and after fresh DNS resolution) and any enabled `intercept.TargetScope` pins/exemptions, dialing only verified addresses. Blocked targets return `0x02` before dialing; DNS lookup failures return `0x04` (*Host unreachable*).

## Raw TCP Storage (SQLite)

Each completed relayed connection writes one row to the `raw_tcp_streams` table in the SQLite traffic index (`traffic/traffic.go`, `indexSchema` — no PostgreSQL migration):

- Columns: `id` (`tcp-<ms>-<seq>`), `protocol` (`'tcp'`), `host`, `port`, `started_at`, `ended_at`, `duration_ms`, `c2s_bytes`, `s2c_bytes`, `truncated`, `c2s_preview`, `c2s_blob`, `s2c_preview`, `s2c_blob`.
- **No HTTP exchange row**: raw TCP streams never insert into `exchanges`.
- **Preview & blob spilling**:
- Directional payloads up to `PreviewBytes` (default `8 KiB`) stay inline in `c2s_preview` / `s2c_preview`.
- Payloads above `PreviewBytes` keep the leading `PreviewBytes` inline and spill the captured bytes (up to `MaxBytesPerDir`) atomically (temp file + rename) into `_blobs/sha256/` with references registered in `blob_refs` under the stream's `tcp-*` ID.
- Exceeding `MaxBytesPerDir` sets `truncated = 1` while continuing to count exact total relayed `c2s_bytes` / `s2c_bytes`.

## Operational Bounds

| Parameter | Default |
|---|---|
| `MaxConcurrent` | `64` active streams (excess connections receive `0x01`) |
| `MaxBytesPerDir` | `4 MiB` captured per direction |
| `PreviewBytes` | `8 KiB` inline preview per direction |
| `IdleTimeout` | `2m` per-direction read/write deadline |
| `HandshakeTimeout` | `10s` for greeting, auth, and `CONNECT` request |
| `DialTimeout` | `30s` outbound TCP connect timeout |
| `PaceTimeout` | `30s` egress governor wait bound |

Closing `Traffic` stops accepting new SOCKS5 connections, closes active sockets, and waits for all handler goroutines to exit before closing SQLite.
59 changes: 59 additions & 0 deletions intercept/target_dial.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,15 @@ package intercept
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"net/netip"
"strconv"
"strings"
"time"

"github.com/Autumn-27/artex/db"
)

// The HTTP clients the platform owns (WebFetch and the custom http tool)
Expand Down Expand Up @@ -178,3 +181,59 @@ func (e *dialError) Error() string {
}
return msg + " Do not fetch the target with another tool or by its address."
}

// IsTargetBlocked reports whether err came from PinnedDialer / DialSOCKS5
// refusing a target before dialing (rule block, unpinned target, or unavailable
// target gate), returning its platform control code.
func IsTargetBlocked(err error) (string, bool) {
var de *dialError
if errors.As(err, &de) {
return de.code, true
}
return "", false
}

// DialSOCKS5 enforces the platform's target gate before dialing a SOCKS5
// CONNECT destination, reusing the same policy functions as PinnedDialer:
// - When ctx carries an enabled TargetScope, delegates directly to dialTarget
// (asset block rules, exemptions, fresh DNS resolution, and pin checks).
// - Otherwise enforces global asset block rules (domain, IP, and CIDR, before
// and after fresh DNS resolution) and dials only the resolved addresses that
// passed those rules.
func (i *Interceptor) DialSOCKS5(ctx context.Context, network, addr string) (net.Conn, error) {
if i == nil {
return baseDial(ctx, network, addr)
}
if scope, ok := TargetScopeFromContext(ctx); ok && scope.Enabled != nil && scope.Enabled() {
return i.dialTarget(ctx, scope, "socks5", network, addr)
}
t, err := addrTarget(network, addr)
if err != nil {
return nil, &dialError{code: TargetGateUnavailable, target: addr, reason: "the address of the connection could not be read (" + err.Error() + ")"}
}
if i.targets == nil {
return i.baseDialer()(ctx, network, addr)
}
rules, err := i.targets.BlockRules(db.TargetScopeRef{})
if err != nil {
return nil, &dialError{code: TargetGateUnavailable, target: t.String(), reason: "the asset rules could not be read: " + err.Error()}
}
ct := &checkedTarget{Target: t}
if t.Kind != TargetDomain {
ct.ips, ct.resolver, ct.resolvedAt = []string{t.Host}, literalResolver, time.Now().UTC()
}
if rule, hit := blockRule(rules, ct); hit {
return nil, &dialError{code: TargetRuleBlocked, target: t.String(), reason: fmt.Sprintf("%s is blocked by an asset rule. %s", t, rule.Reason())}
}
if t.Kind == TargetDomain {
c := &targetCheck{i: i, ctx: ctx, tool: "socks5", fresh: true}
c.resolve(ct, true)
if ct.err != nil {
return nil, ct.err
}
if rule, hit := blockRule(rules, ct); hit {
return nil, &dialError{code: TargetRuleBlocked, target: t.String(), reason: fmt.Sprintf("%s is blocked by an asset rule. %s", t, rule.Reason())}
}
}
return i.dialAddrs(ctx, network, ct.ips, t.Port)
}
8 changes: 8 additions & 0 deletions server/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,14 @@ func NewManager(dir, proxyAddr string) (*Manager, error) {
if tr != nil {
m.traffic = tr
m.reloadRewriteRules() // in place before the proxy accepts a connection
if scfg, ok := traffic.SOCKS5ConfigFromEnv(); ok {
scfg.DialContext = m.interceptor.DialSOCKS5
if bound, serr := tr.StartSOCKS5(scfg); serr != nil {
log.Printf("[traffic/socks5] disabled: %v", serr)
} else {
log.Printf("[traffic/socks5] opt-in SOCKS5 CONNECT listener on %s", bound)
}
}
go func() {
log.Printf("[traffic] recording proxy on %s (set HTTP_PROXY=%s + trust _ca CA)", proxyAddr, tr.ProxyAddr())
if err := tr.Start(); err != nil {
Expand Down
108 changes: 108 additions & 0 deletions server/socks5_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
package server

import (
"encoding/binary"
"fmt"
"io"
"net"
"strconv"
"sync/atomic"
"testing"
"time"

"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/traffic"
)

func TestManagerSOCKS5OptInFlagAndAssetBlockRuleRefusal(t *testing.T) {
// 1. Flag off (default): Manager starts NO SOCKS5 listener.
t.Setenv(traffic.SOCKS5EnabledEnv, "")
t.Setenv(traffic.SOCKS5AddrEnv, "")
mOff, err := NewManager(t.TempDir(), "127.0.0.1:0")
if err != nil {
db.SkipPGUnavailable(t, err)
}
t.Cleanup(func() { _ = mOff.Close() })
if mOff.traffic == nil {
t.Fatal("expected HTTP recording proxy to start")
}
if got := mOff.traffic.SOCKS5Addr(); got != "" {
t.Fatalf("SOCKS5Addr with flag off = %q, want empty", got)
}

// 2. Flag on (ARTEX_SOCKS5_ADDR=127.0.0.1:0): listener starts and enforces
// global asset block rules via m.interceptor.DialSOCKS5 BEFORE dialing.
t.Setenv(traffic.SOCKS5AddrEnv, "127.0.0.1:0")
mOn, err := NewManager(t.TempDir(), "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = mOn.Close() })
socksAddr := mOn.traffic.SOCKS5Addr()
if socksAddr == "" {
t.Fatal("expected SOCKS5 listener when ARTEX_SOCKS5_ADDR is set")
}

// Add a global asset block rule for 127.0.0.1 and ensure cleanup.
rule, err := mOn.pg.CreateAssetInterceptRule(
"exact_ip",
"127.0.0.1",
fmt.Sprintf("p41a-block-test-%d", time.Now().UnixNano()),
true,
)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = mOn.pg.DeleteAssetInterceptRule(rule.ID) })

// Target TCP server on 127.0.0.1: must NEVER receive a connection.
targetLn, err := net.Listen("tcp4", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer targetLn.Close()
var accepted atomic.Int32
go func() {
for {
c, err := targetLn.Accept()
if err != nil {
return
}
accepted.Add(1)
_ = c.Close()
}
}()
_, portStr, _ := net.SplitHostPort(targetLn.Addr().String())
port, _ := strconv.Atoi(portStr)

c, err := net.DialTimeout("tcp", socksAddr, 3*time.Second)
if err != nil {
t.Fatal(err)
}
defer c.Close()
_ = c.SetDeadline(time.Now().Add(5 * time.Second))
// Greeting: VER=5, NMETHODS=1, NO AUTH
if _, err := c.Write([]byte{0x05, 0x01, 0x00}); err != nil {
t.Fatal(err)
}
var sel [2]byte
if _, err := io.ReadFull(c, sel[:]); err != nil || sel != [2]byte{0x05, 0x00} {
t.Fatalf("method selection = %v err=%v", sel, err)
}
// CONNECT 127.0.0.1:<port>
req := []byte{0x05, 0x01, 0x00, 0x01, 127, 0, 0, 1, 0, 0}
binary.BigEndian.PutUint16(req[8:10], uint16(port))
if _, err := c.Write(req); err != nil {
t.Fatal(err)
}
var rep [10]byte
if _, err := io.ReadFull(c, rep[:]); err != nil {
t.Fatal(err)
}
if rep[1] != 0x02 {
t.Fatalf("blocked target SOCKS5 reply = 0x%02x, want 0x02 (not allowed by ruleset)", rep[1])
}
if accepted.Load() != 0 {
t.Fatalf("blocked target accepted %d connections, want 0", accepted.Load())
}
}
Loading
Loading