Add Harbor mode: run SIA self-improvement on external benchmarks

README.md

@@ -217,6 +217,36 @@ Full step-by-step for both paths: [docs/walkthrough.md](docs/walkthrough.md).
 
 ---
 
+## Run on a Harbor benchmark
+
+Instead of a local task, SIA can self-improve against any external benchmark from the
+[Harbor](https://www.harborframework.com/docs) registry (Terminal-Bench, SWE-Bench, AIME, …). Each
+generation's agent is run **inside the benchmark's own Docker containers** and scored by Harbor's
+native verifiers — no local `evaluate.py` required.
+
+```bash
+uv tool install harbor          # the Harbor CLI (ships its own interpreter)
+harbor auth login               # one-time
+pip install 'sia-agent[claude,harbor]'
+export ANTHROPIC_API_KEY="..."
+
+sia run --harbor_dataset terminal-bench-sample@2.0 --max_gen 3 --run_id 1
+```
+
+| Flag | Description |
+|---|---|
+| `--harbor_dataset NAME@VERSION` | Benchmark to download from the Harbor registry (enables Harbor mode) |
+| `--harbor_task_dir PATH` | Use a pre-downloaded benchmark directory instead |
+| `--harbor_include_task NAME` | Restrict the run to specific task(s); repeatable (cheap for testing) |
+| `--harbor_working_dir DIR` | Container working directory the agent operates in (default `/app`) |
+
+In Harbor mode the meta/feedback prompts gain an injected in-container contract (the base prompts are
+unchanged): the generated `target_agent.py` takes `--working_dir / --instruction_file / --log_dir`,
+runs an agentic bash loop inside the container, and leaves it in the state the verifier expects. See
+[docs/harbor.md](docs/harbor.md) for architecture and details.
+
+---
+
 ## Evaluation
 
 After every generation the orchestrator scores the target agent automatically and
@@ -247,6 +277,7 @@ Full contract, return-format rules, and a complete example: [EVALUATION_GUIDE.md
 
 - [docs/architecture.md](docs/architecture.md) — directory layout, generation flow, prompt customization
 - [docs/walkthrough.md](docs/walkthrough.md) — detailed custom-task walkthrough
+- [docs/harbor.md](docs/harbor.md) — run SIA against external Harbor benchmarks
 - [docs/configuration.md](docs/configuration.md) — agent impls, models, API keys, CLI reference
 - [EVALUATION_GUIDE.md](EVALUATION_GUIDE.md) — writing `evaluate.py` for a custom task
 - [docs/troubleshooting.md](docs/troubleshooting.md) — common errors and fixes

docs/harbor.md

@@ -0,0 +1,91 @@
+# Running SIA on Harbor benchmarks
+
+SIA can run its self-improvement loop against external benchmarks from the
+[Harbor](https://www.harborframework.com/docs) registry. Each generation's `target_agent.py` is
+attached to the benchmark, executed **inside the benchmark's own Docker containers**, and scored by
+Harbor's **native verifiers** — there is no local dataset or `evaluate.py`.
+
+## How it maps onto SIA
+
+| SIA concept | Local mode | Harbor mode |
+|---|---|---|
+| Task | a `tasks/<name>/` directory | a downloaded Harbor benchmark (many container tasks) |
+| Running a generation | `python target_agent.py --dataset_dir … --working_dir …` | one Harbor **job**: the agent runs in every task's container |
+| Scoring | local `evaluate.py` → `results.json` | each task's verifier → reward → aggregated `results.json` |
+| Feedback input | execution log + `results.json` | per-task trajectories + per-task rewards (same files) |
+
+The self-improvement loop itself is unchanged: meta-agent writes gen 1, the feedback agent reads the
+scores + trajectories and rewrites the agent for gen 2, and so on.
+
+## Quick start
+
+```bash
+uv tool install harbor          # Harbor CLI (isolated; SIA calls it as a subprocess)
+harbor auth login               # one-time, for the registry
+pip install 'sia-agent[claude,harbor]'
+export ANTHROPIC_API_KEY="..."
+
+# Cheap smoke test: one tiny task
+sia run --harbor_dataset terminal-bench-sample@2.0 --harbor_include_task log-summary-date-ranges \
+    --max_gen 2 --run_id 1
+
+# Full benchmark
+sia run --harbor_dataset terminal-bench-sample@2.0 --max_gen 3 --run_id 2
+```
+
+## Flags
+
+| Flag | Description |
+|---|---|
+| `--harbor_dataset NAME@VERSION` | Benchmark downloaded from the Harbor registry (enables Harbor mode) |
+| `--harbor_task_dir PATH` | Use a pre-downloaded benchmark directory (a dir whose children are task folders) |
+| `--harbor_include_task NAME` | Restrict to specific task name(s); repeatable. Keeps test runs cheap. |
+| `--harbor_working_dir DIR` | Container working directory the agent operates in (default `/app`) |
+
+`SIA_HARBOR_BIN` overrides the path to the `harbor` CLI if it is not on `PATH`.
+
+## The in-container agent contract
+
+In Harbor mode the meta/feedback prompts are **appended** (not edited) with an in-container contract.
+The generated `target_agent.py` must follow:
+
+```
+python3 target_agent.py --working_dir <dir> --instruction_file <path> --log_dir <dir>
+```
+
+Inside the container it is guaranteed: internet access, an LLM API key in the environment, the model
+id in `SIA_TASK_MODEL`, and the `anthropic` SDK installed. The agent runs an agentic bash loop —
+explore the working dir, edit files, verify — and leaves the container in the state the task's
+verifier checks. It must **not** write a submission file. The bundled reference implementation is
+[`sia/tasks/_shared/reference_harbor_agent.py`](../sia/tasks/_shared/reference_harbor_agent.py); the
+meta-agent models its output on it.
+
+## Architecture
+
+Two small modules implement the bridge, with no changes to the agent harness:
+
+- **`sia/harbor_agent.py`** — `SIATargetAgent`, a Harbor `BaseAgent`. For each task it uploads the
+  generation's `target_agent.py` into the container, runs it against the task instruction, downloads
+  the trajectory back to the host, and lets the task's verifier score the container.
+- **`sia/harbor_runner.py`** — drives the `harbor` CLI as a subprocess (Harbor ships its own
+  interpreter), then parses the job's `result.json` into SIA's `results.json` plus per-task
+  `agent_execution/execution_q{i}.json` trajectories that the feedback agent already understands.
+
+## Output
+
+Per generation, under `runs/run_{id}/gen_{n}/`:
+
+- `target_agent.py` — the agent for that generation
+- `results.json` — `{score, mean_reward, n_tasks, n_solved, n_errors, per_task: [...]}`
+- `agent_execution/execution_q{i}.json` — one trajectory per benchmark task
+- `harbor_jobs/` — the raw Harbor job output (per-trial `result.json`, verifier reward, logs)
+- `harbor_run.log` — stdout/stderr from the Harbor CLI
+
+## Notes & limits
+
+- Benchmark images must allow internet (the default) so the agent can install the SDK and call the
+  model. The adapter installs `anthropic` best-effort; on a base image without Python it `apt`-installs
+  it first.
+- A task that scores `0` is still a valid run — that is exactly the signal the feedback agent uses to
+  improve the next generation.
+- Harbor removes task containers/images after each run, so every run re-pulls images.

pyproject.toml

@@ -37,6 +37,10 @@ claude = ["claude-agent-sdk>=0.1.50"]
 openhands = ["openhands-ai>=1.6.0"]
 pydantic-ai = ["pydantic-ai>=1.0"]
 mlebench = ["google-generativeai>=0.8"]
+# Harbor mode drives the `harbor` CLI as a subprocess (it ships its own interpreter),
+# so no Python dependency is pinned here. Install the CLI separately, e.g.
+# `uv tool install harbor`, or set SIA_HARBOR_BIN to its path.
+harbor = []
 dev = [
     "pytest>=7.0",
     "ruff>=0.11",

sia/cli.py

@@ -39,7 +39,7 @@ def _add_run_args(parser: argparse.ArgumentParser, env_config: Config) -> None:
         help="Maximum number of generations to run (default: 3)",
     )
     parser.add_argument("--run_id", type=int, default=1, help="Run ID for this experiment (default: 1)")
-    task_group = parser.add_mutually_exclusive_group(required=True)
+    task_group = parser.add_mutually_exclusive_group(required=False)
     task_group.add_argument(
         "--task",
         type=str,
@@ -51,6 +51,31 @@ def _add_run_args(parser: argparse.ArgumentParser, env_config: Config) -> None:
         type=str,
         help="Path to an external task directory (e.g., ./tasks/my-task)",
     )
+    parser.add_argument(
+        "--harbor_dataset",
+        type=str,
+        default=None,
+        help="Harbor benchmark to run on, e.g. 'terminal-bench-sample@2.0' (enables Harbor mode)",
+    )
+    parser.add_argument(
+        "--harbor_task_dir",
+        type=str,
+        default=None,
+        help="Path to a pre-downloaded Harbor benchmark directory (enables Harbor mode)",
+    )
+    parser.add_argument(
+        "--harbor_include_task",
+        action="append",
+        default=None,
+        metavar="TASK_NAME",
+        help="Restrict the Harbor run to these task name(s); repeatable",
+    )
+    parser.add_argument(
+        "--harbor_working_dir",
+        type=str,
+        default="/app",
+        help="Container working directory the agent operates in (default: /app)",
+    )
     parser.add_argument(
         "--meta-agent-profile",
         dest="meta_agent_profile",

sia/harbor_agent.py

@@ -0,0 +1,115 @@
+"""Harbor agent that runs a SIA-generated target agent inside a benchmark task container."""
+
+import json
+import os
+import shlex
+from pathlib import Path
+
+from harbor.agents.base import BaseAgent
+from harbor.environments.base import BaseEnvironment
+from harbor.models.agent.context import AgentContext
+
+FORWARDED_KEY_VARS = (
+    "ANTHROPIC_API_KEY",
+    "GEMINI_API_KEY",
+    "GOOGLE_API_KEY",
+    "OPENAI_API_KEY",
+)
+AGENT_DIR = "/tmp/sia_agent"
+OUTPUT_DIR = "/tmp/sia_agent/out"
+
+
+async def _maybe_await(value):
+    return await value if hasattr(value, "__await__") else value
+
+
+class SIATargetAgent(BaseAgent):
+    def __init__(
+        self,
+        *args,
+        agent_path: str | None = None,
+        working_dir: str = "/app",
+        task_model: str | None = None,
+        run_timeout: int = 900,
+        pip_packages: str = "anthropic",
+        **kwargs,
+    ):
+        super().__init__(*args, **kwargs)
+        if not agent_path:
+            raise ValueError("SIATargetAgent requires agent_path (the generated target_agent.py)")
+        self._agent_path = str(Path(agent_path).expanduser().resolve())
+        self._working_dir = working_dir
+        self._task_model = task_model or self.model_name
+        self._run_timeout = int(run_timeout)
+        self._pip_packages = pip_packages.strip()
+
+    @staticmethod
+    def name() -> str:
+        return "sia-target-agent"
+
+    def version(self) -> str:
+        return "0.1.0"
+
+    def _agent_env(self) -> dict:
+        env = {k: os.environ[k] for k in FORWARDED_KEY_VARS if os.environ.get(k)}
+        if self._task_model:
+            env["SIA_TASK_MODEL"] = self._task_model
+        return env
+
+    async def setup(self, environment: BaseEnvironment) -> None:
+        await _maybe_await(
+            environment.exec(
+                "command -v python3 >/dev/null 2>&1 || (apt-get update && apt-get install -y python3 python3-pip)",
+                timeout_sec=600,
+            )
+        )
+        if self._pip_packages:
+            await _maybe_await(
+                environment.exec(
+                    f"python3 -m pip install --quiet {self._pip_packages} --break-system-packages "
+                    f"|| python3 -m pip install --quiet {self._pip_packages}",
+                    timeout_sec=600,
+                )
+            )
+        await _maybe_await(environment.exec(f"mkdir -p {AGENT_DIR} {OUTPUT_DIR}"))
+
+    async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
+        await _maybe_await(environment.upload_file(self._agent_path, f"{AGENT_DIR}/target_agent.py"))
+        instruction_host = Path(self.logs_dir) / "INSTRUCTION.md"
+        instruction_host.write_text(instruction, encoding="utf-8")
+        await _maybe_await(environment.upload_file(str(instruction_host), f"{AGENT_DIR}/INSTRUCTION.md"))
+
+        cmd = (
+            f"cd {shlex.quote(self._working_dir)} && "
+            f"python3 {AGENT_DIR}/target_agent.py "
+            f"--working_dir {shlex.quote(self._working_dir)} "
+            f"--instruction_file {AGENT_DIR}/INSTRUCTION.md "
+            f"--log_dir {OUTPUT_DIR}"
+        )
+        result = await _maybe_await(environment.exec(cmd, env=self._agent_env(), timeout_sec=self._run_timeout))
+
+        stdout = getattr(result, "stdout", "") or ""
+        stderr = getattr(result, "stderr", "") or ""
+        return_code = getattr(result, "return_code", None)
+        (Path(self.logs_dir) / "agent_stdout.log").write_text(stdout + "\n" + stderr, encoding="utf-8")
+
+        try:
+            await _maybe_await(environment.download_dir(OUTPUT_DIR, str(Path(self.logs_dir) / "sia_out")))
+        except Exception as exc:
+            (Path(self.logs_dir) / "download_error.txt").write_text(str(exc), encoding="utf-8")
+
+        self._populate_context(context, return_code)
+
+    def _populate_context(self, context: AgentContext, return_code) -> None:
+        traj = Path(self.logs_dir) / "sia_out" / "agent_execution.json"
+        if not traj.is_file():
+            return
+        try:
+            usage = json.loads(traj.read_text(encoding="utf-8")).get("usage", {})
+        except Exception:
+            return
+        if usage.get("input_tokens"):
+            context.n_input_tokens = usage["input_tokens"]
+        if usage.get("output_tokens"):
+            context.n_output_tokens = usage["output_tokens"]
+        context.metadata = {**(context.metadata or {}), "agent_return_code": return_code}