#38: Persist restricted-egress enforcement evidence

docs/lab-runtime-hardening.md

@@ -86,21 +86,28 @@ The current Kali lab container is appropriate for bounded static analysis, passi
 
 When a network profile has egress rules defined, the lab runtime:
 
-1. Starts a dedicated egress-controller sidecar in the lab's network namespace.
-2. Grants `NET_ADMIN` only to that sidecar; the workload retains `--cap-drop ALL` and never receives `NET_ADMIN` or `NET_RAW`.
-3. Applies the iptables policy through the controller, then records the network-profile and a SHA-256 policy fingerprint as Docker labels on the controller.
-4. The script sets default DROP policies on INPUT, FORWARD, and OUTPUT chains, then adds ACCEPT rules for:
+1. Creates the workload with `--network none`, starts it without a network, pauses it, and only then attaches the approved Docker network. This prevents the image entrypoint from having an unfiltered startup window.
+2. Starts a dedicated egress-controller sidecar in the paused lab's network namespace.
+3. Grants `NET_ADMIN` only to that sidecar; the workload retains `--cap-drop ALL` and never receives `NET_ADMIN` or `NET_RAW`.
+4. Applies the iptables policy through the controller, then records the network-profile and a SHA-256 policy fingerprint as Docker labels on the controller.
+5. The script sets default DROP policies on IPv4 and IPv6 INPUT, FORWARD, and OUTPUT chains. If IPv6 is active but `ip6tables` is unavailable, admission fails closed. It then adds IPv4 ACCEPT rules for:
    - Established/related connections (so response traffic is allowed).
    - Loopback traffic (localhost communication within the container).
    - Each egress rule destination:port/protocol.
-4. A final REJECT rule on OUTPUT drops all other outbound traffic.
+6. A final REJECT rule on OUTPUT drops all other outbound traffic.
 
 ### Approved-targets enforcement
 
 For the `approved-targets` profile, the iptables script is dynamically built from the approved target list. Each approved target is resolved to a hostname and port (443 for HTTPS, 80 for HTTP), and an ACCEPT rule is added for that destination. No other outbound traffic is allowed.
 
 Denied packets are rate-limited and logged with the `EXPLOIT_HUNTER_EGRESS_DENIED` prefix. The controller lifecycle, policy fingerprint, and Docker command trace are inspectable forensic evidence; workloads cannot modify the firewall because they do not possess the capability.
 
+### Durable network evidence
+
+Lab start and restart now save the external controller's enforcement result through the central Artifact service as project-scoped JSONL. A durable database receipt is admitted first; artifact delivery failure leaves a retryable failed receipt and prevents the restricted workload from becoming `running`. Each record uses the versioned `exploit-hunter.lab-network-evidence.v1` schema and can carry project, thread, task, target, tool-run, research-run, network-profile, and full policy-digest correlation. Enforcement failures are recorded as `enforcement-unavailable`; successful policy installation is recorded separately as `policy-enforced` and is not represented as proof that a connection was allowed.
+
+Before a network-capable approved command starts, execution admits an intent receipt bound to the server-owned tool run and current policy snapshot. After execution, deltas from the controller's packet counters are persisted as grounded allowed/denied policy decisions; shell text and exit status are not treated as network facts. A failed finalization leaves the pre-execution receipt retryable and does not invite replay of the command. Counter deltas are not a packet-complete transcript and may not identify every destination. Controller observations must retain the same project and full policy digest through finalization. `pnpm labs:repair-evidence` retries artifact delivery and orphan-runtime cleanup receipts.
+
 ### Package-egress enforcement
 
 For the `package-egress` profile, the iptables script allows traffic only to well-known package registries and distribution mirrors. This covers npm, Yarn, PyPI, RubyGems, crates.io, GitHub release objects, Debian/Ubuntu apt repositories, and Docker Hub.
@@ -119,7 +126,7 @@ Network profile changes are bound to durable approvals. The `agentLabCommandTool
 
 ## Limitations
 
-- iptables rules are applied after container start and are lost on container restart. The external controller re-applies rules on every start.
+- iptables rules are lost on container recreation. Restricted workloads are recreated offline and paused while the external controller reapplies and verifies the rules on every start.
 - DNS resolution for approved targets uses the container's configured DNS resolver. The iptables rules match against resolved IP addresses at connection time, not domain names. This means the hostname-based allowlisting is resolved at the time each connection is made.
 - The firewall controller shares the lab network namespace, but the workload has no firewall administration capability. A host-level or dedicated-network firewall remains the stronger option against a Docker daemon or kernel compromise.
 - Namespace/cgroup isolation shares the host kernel with the lab workload. For labs where a kernel-level container escape is an unacceptable risk (for example, running untrusted exploit code or malware samples), use `microvm` isolation (see above) so the lab runs in its own guest kernel instead of the host's.

package.json

@@ -26,6 +26,7 @@
     "db:reset": "tsx scripts/db-reset.ts",
     "db:new": "tsx scripts/db-new-migration.ts",
     "db:seed": "tsx scripts/db-seed.ts",
+    "labs:repair-evidence": "tsx scripts/repair-lab-evidence.ts",
     "setup": "tsx scripts/setup.ts",
     "embeddings:init": "tsx scripts/init-embedding-runtime.ts",
     "rag:reindex": "tsx scripts/reindex-rag.ts",

scripts/repair-lab-evidence.ts

@@ -0,0 +1,13 @@
+import { retryLabRuntimeCleanupReceipts } from "../src/server/labs/cleanup-receipts";
+import { retryLabNetworkEvidenceReceipts } from "../src/server/labs/network-evidence";
+
+const limitArg = process.argv.find((value) => value.startsWith("--limit="));
+const parsedLimit = limitArg ? Number(limitArg.slice("--limit=".length)) : 25;
+const limit = Number.isFinite(parsedLimit) ? parsedLimit : 25;
+
+const [evidence, cleanup] = await Promise.all([
+  retryLabNetworkEvidenceReceipts({ limit }),
+  retryLabRuntimeCleanupReceipts({ limit }),
+]);
+
+process.stdout.write(`${JSON.stringify({ evidence, cleanup })}\n`);

src/lib/ids.ts

@@ -35,6 +35,8 @@ export type AppIdKind =
   | "message"
   | "modelConfig"
   | "negativeResult"
+  | "networkEvidenceReceipt"
+  | "labRuntimeCleanup"
   | "plan"
   | "project"
   | "queuedMessage"
@@ -93,6 +95,8 @@ const ID_SPECS = {
   message: { prefix: "msg", length: HIGH_CHURN_ID_LENGTH },
   modelConfig: { prefix: "mdl" },
   negativeResult: { prefix: "neg" },
+  networkEvidenceReceipt: { prefix: "ner", length: HIGH_CHURN_ID_LENGTH },
+  labRuntimeCleanup: { prefix: "lrc", length: HIGH_CHURN_ID_LENGTH },
   plan: { prefix: "pln" },
   project: { prefix: "prj" },
   queuedMessage: { prefix: "que" },

src/mastra/tools/agent-lab-command.ts

@@ -131,18 +131,20 @@ const commandEnvSchema = z.preprocess((value) => {
   }
   return value;
 }, z.array(commandEnvEntrySchema).optional());
-const commandArgsSchema = z.preprocess(
-  (value) =>
-    typeof value === "string"
-      ? value
-          .split(",")
-          .map((entry) => entry.trim())
-          .filter(Boolean)
-      : value,
-  z.array(z.string()).optional(),
-).describe(
-  "Command arguments as separate array entries. For inline interpreters, pass the executable in command and the script flag/source here, for example command: python3 with args: [-c, <source>].",
-);
+const commandArgsSchema = z
+  .preprocess(
+    (value) =>
+      typeof value === "string"
+        ? value
+            .split(",")
+            .map((entry) => entry.trim())
+            .filter(Boolean)
+        : value,
+    z.array(z.string()).optional(),
+  )
+  .describe(
+    "Command arguments as separate array entries. For inline interpreters, pass the executable in command and the script flag/source here, for example command: python3 with args: [-c, <source>].",
+  );
 const optionalNumberSchema = z.preprocess(
   (value) => (typeof value === "string" && value.trim() ? Number(value) : value),
   z.number().optional(),
@@ -879,6 +881,9 @@ const agentLabCommandToolDefinition = {
         shellWorkspacePath: executionBinding.workspacePath,
         agentAccessEnabled: true,
         toolRunId: queuedToolRun.id,
+        taskId,
+        targetIds,
+        researchRunId: readStringValue(contextValue(context, "researchRunId")),
         onChunk: async (chunk) => {
           for (const data of limitStreamChunk(chunk)) {
             await emitLabCommandStreamEvent(context, {

src/server/agent-lab/command-runner.ts

@@ -1,5 +1,6 @@
-import { spawn } from "node:child_process";
+import { execFile, spawn } from "node:child_process";
 import { randomUUID } from "node:crypto";
+import { promisify } from "node:util";
 
 import type {
   CommandResult,
@@ -13,7 +14,11 @@ import {
   rewriteRemoteWorkspaceCommand,
 } from "../compute";
 import { labContainerIdentity } from "../labs/docker-plan";
-import { getProjectLabStatus } from "../labs/service";
+import {
+  completeProjectLabNetworkIntent,
+  getProjectLabStatus,
+  openProjectLabNetworkIntent,
+} from "../labs/service";
 import {
   getLabSharedTerminalRegistry,
   getLabSshSharedTerminalRegistry,
@@ -23,6 +28,8 @@ import { LEAD_AGENT_ID, sharedTerminalSessionId } from "../terminal/shared-sessi
 import { getDefaultThreadWorkspaceService } from "../workspaces";
 import type { ThreadTargetConfig } from "../workspaces/target-mode";
 
+const execFileAsync = promisify(execFile);
+
 export type AgentLabCommandStreamChunk = {
   stream: "stdout" | "stderr";
   data: string;
@@ -88,6 +95,9 @@ export interface ProjectAgentLabCommandRunInput {
   shellWorkspacePath?: string;
   agentAccessEnabled?: boolean;
   toolRunId?: string;
+  taskId?: string;
+  targetIds?: string[];
+  researchRunId?: string;
 }
 
 export interface ProjectAgentLabCommandRunner {
@@ -402,6 +412,9 @@ export const projectAgentLabCommandRunner: ProjectAgentLabCommandRunner = {
     shellWorkspacePath = "/workspace",
     agentAccessEnabled = true,
     toolRunId,
+    taskId,
+    targetIds,
+    researchRunId,
   }) {
     if (target.targetMode === "none") {
       throw new Error("Command execution is disabled for this thread workspace targetMode.");
@@ -526,6 +539,23 @@ export const projectAgentLabCommandRunner: ProjectAgentLabCommandRunner = {
         agentId,
         agentAccessEnabled,
       });
+      const networkCapable = commandMayUseNetwork(fullCommand);
+      const networkIntent = networkCapable
+        ? await openProjectLabNetworkIntent({
+            projectId,
+            threadId,
+            taskId,
+            targetIds,
+            toolRunId,
+            researchRunId,
+            command: fullCommand,
+          })
+        : undefined;
+      const beforeCounters = networkIntent
+        ? await captureEgressControllerCounters(containerName, networkIntent.policyId).catch(
+            () => undefined,
+          )
+        : undefined;
       const commandResult = await session.runCommandTurn({
         actor: `agent:${agentId}`,
         command: buildSharedTerminalCommand(fullCommand, cwd, commandEnv(env)),
@@ -559,13 +589,108 @@ export const projectAgentLabCommandRunner: ProjectAgentLabCommandRunner = {
       };
       safeToReleaseWorkspace =
         !commandResult.timedOut || commandResult.termination?.confirmed === true;
+      if (networkIntent) {
+        const afterCounters = await captureEgressControllerCounters(
+          containerName,
+          networkIntent.policyId,
+        ).catch(() => undefined);
+        const networkObservations = networkObservationsFromControllerCounters(
+          beforeCounters,
+          afterCounters,
+        );
+        await completeProjectLabNetworkIntent({
+          intent: networkIntent,
+          projectId,
+          threadId,
+          taskId,
+          targetIds,
+          toolRunId,
+          researchRunId,
+          observations:
+            networkObservations.length > 0
+              ? networkObservations
+              : [
+                  {
+                    observedAt: new Date().toISOString(),
+                    event: "enforcement-state",
+                    disposition: "policy-enforced",
+                    reason: "No attributable egress-controller counter delta was observed.",
+                    source: "external-egress-controller-counters",
+                  },
+                ],
+        }).catch((error) => {
+          console.error(
+            `[agent-lab] Network evidence finalization is pending repair for ${networkIntent.receiptId}: ${error instanceof Error ? error.message : String(error)}`,
+          );
+        });
+      }
       return output;
     } finally {
       await workspaceLock.release(safeToReleaseWorkspace);
     }
   },
 };
 
+export const commandMayUseNetwork = (command: string) =>
+  /(?:\bhttps?:\/\/|\b(?:curl|wget|nc|ncat|netcat|dig|host|nslookup|nmap|masscan|naabu|ssh|scp|sftp|ftp|telnet|openssl\s+s_client)\b)/i.test(
+    command,
+  );
+
+type EgressCounterSnapshot = {
+  policyId: string;
+  capturedAt: string;
+  rules: Map<string, { packets: number; target: string }>;
+};
+
+async function captureEgressControllerCounters(
+  workloadContainer: string,
+  policyId: string,
+): Promise<EgressCounterSnapshot> {
+  const result = await execFileAsync("docker", [
+    "exec",
+    `${workloadContainer}-egress-enforcer`,
+    "iptables-save",
+    "-c",
+    "-t",
+    "filter",
+  ]);
+  return parseEgressControllerCounters(result.stdout, policyId);
+}
+
+export function parseEgressControllerCounters(
+  text: string,
+  policyId: string,
+): EgressCounterSnapshot {
+  const rules = new Map<string, { packets: number; target: string }>();
+  for (const line of text.split("\n")) {
+    const match = /^\[(\d+):\d+\]\s+(-A OUTPUT.*-j (?:ACCEPT|REJECT))$/.exec(line.trim());
+    if (!match) continue;
+    rules.set(match[2]!, { packets: Number(match[1]), target: match[2]! });
+  }
+  return { policyId, capturedAt: new Date().toISOString(), rules };
+}
+
+export function networkObservationsFromControllerCounters(
+  before: EgressCounterSnapshot | undefined,
+  after: EgressCounterSnapshot | undefined,
+): import("../labs/network-evidence").LabNetworkObservation[] {
+  if (!before || !after || before.policyId !== after.policyId) return [];
+  const observations: import("../labs/network-evidence").LabNetworkObservation[] = [];
+  for (const [key, current] of after.rules) {
+    const delta = current.packets - (before.rules.get(key)?.packets ?? 0);
+    if (delta <= 0) continue;
+    observations.push({
+      observedAt: after.capturedAt,
+      event: "policy-counter",
+      disposition: / -j ACCEPT$/.test(current.target) ? "allowed" : "denied",
+      packetCount: delta,
+      reason: current.target,
+      source: "external-egress-controller-counters",
+    });
+  }
+  return observations;
+}
+
 async function acquireCommandWorkspaceLock(projectId: string, threadId: string, timeoutMs: number) {
   const service = getDefaultThreadWorkspaceService();
   const workspace = await service.resolve({ projectId, threadId }).catch(() => undefined);

src/server/db/migrations/20260830180000_network_evidence_receipts.sql

@@ -0,0 +1,22 @@
+-- migrate:up
+CREATE TABLE IF NOT EXISTS network_evidence_receipts (
+  id text PRIMARY KEY,
+  project_id text NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+  thread_id text,
+  policy_id text,
+  status text NOT NULL CHECK (status IN ('pending', 'recorded', 'failed')),
+  payload text NOT NULL,
+  artifact_id text REFERENCES artifacts(id) ON DELETE SET NULL,
+  attempts integer NOT NULL DEFAULT 0,
+  last_error text,
+  created_at text NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  updated_at text NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE INDEX IF NOT EXISTS network_evidence_receipts_delivery_idx
+  ON network_evidence_receipts(status, created_at, id);
+CREATE INDEX IF NOT EXISTS network_evidence_receipts_project_idx
+  ON network_evidence_receipts(project_id, created_at, id);
+
+-- migrate:down
+DROP TABLE IF EXISTS network_evidence_receipts;

src/server/db/migrations/20260830190000_lab_runtime_cleanup_receipts.sql

@@ -0,0 +1,24 @@
+-- migrate:up
+CREATE TABLE IF NOT EXISTS lab_runtime_cleanup_receipts (
+  id text PRIMARY KEY,
+  project_id text NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+  lab_id text NOT NULL REFERENCES project_labs(id) ON DELETE CASCADE,
+  runtime_locator text NOT NULL,
+  status text NOT NULL CHECK (status IN ('pending', 'completed')),
+  payload text NOT NULL,
+  attempts integer NOT NULL DEFAULT 0,
+  last_error_message text,
+  created_at text NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  updated_at text NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE INDEX IF NOT EXISTS lab_runtime_cleanup_delivery_idx
+  ON lab_runtime_cleanup_receipts(status, updated_at, id);
+
+CREATE UNIQUE INDEX IF NOT EXISTS artifacts_network_evidence_receipt_name_idx
+  ON artifacts(project_id, name)
+  WHERE name LIKE 'network-evidence-ner%';
+
+-- migrate:down
+DROP TABLE IF EXISTS lab_runtime_cleanup_receipts;
+DROP INDEX IF EXISTS artifacts_network_evidence_receipt_name_idx;

src/server/db/postgres-migrate.ts

@@ -177,6 +177,8 @@ const APP_TABLES = [
   "policy_trajectory_feedback",
   "passive_policy_shadow_records",
   "blockers",
+  "network_evidence_receipts",
+  "lab_runtime_cleanup_receipts",
 ] as const;
 const POSTGRES_JSON_TEXT_EXCEPTIONS = new Set([
   // This is a constrained workflow enum, despite sharing a legacy SQLite JSON-column name.

src/server/evidence/ingestion.ts

@@ -28,6 +28,7 @@ export const EVIDENCE_SOURCES = [
   "stage-handoff",
   "passive-recon",
   "reference",
+  "network-observation",
 ] as const;
 
 export type EvidenceSource = (typeof EVIDENCE_SOURCES)[number];