diff --git a/README.md b/README.md index 35986fa1..cd42e792 100644 --- a/README.md +++ b/README.md @@ -165,12 +165,36 @@ on a free-tier account; use the fast subset for cost-bounded runs. ## Benchmark integrity -Agents should not be able to look up reference solutions during a run. The -bundled `claude-code` configs disable web tools -(`disallowed_tools: WebSearch,WebFetch`); disable the equivalent browsing tools -for other agents. For stricter isolation, run under Harbor's network allowlist -with `--allow-agent-host`: the DuckDB variant needs only your model API host, -and the Snowflake variant also needs `.snowflakecomputing.com`. +Agents should not be able to look up reference solutions during a run. This +matters more now that the tasks and their reference solutions are public. + +The bundled configs disable web tools where the agent supports it: + +| Config | Web tools | +|---|---| +| `claude-code` | off — `disallowed_tools: WebSearch,WebFetch` | +| `codex` | off — `web_search: disabled` | +| `cortex-code` | off — `disallowed_tools: web_search web_fetch` (requires Harbor >= 0.22.0) | +| `terminus-2` (the commented fallback) | none exist | + +Cortex Code keeps `web_search` and `web_fetch` available in every agent mode, +including code mode. Harbor's `cortex-code` agent only gained the option to +switch them off in [harbor#2787](https://github.com/harbor-framework/harbor/pull/2787), +which shipped in Harbor 0.22.0. On an older Harbor the kwarg is dropped +silently rather than rejected, so if you're not on 0.22.0+, isolate +`cortex-code` runs at the network layer instead (below). + +For any agent, Harbor's network allowlist is the stricter control, and the only +one that does not depend on agent support: + +```bash +harbor run ... --allow-agent-host +``` + +The DuckDB variant needs only your model API host; the Snowflake variant also +needs `.snowflakecomputing.com`. If you add a config for another agent, +disable its browsing tools too — and check that the setting took effect, since +an unrecognised kwarg is dropped silently rather than rejected. ## Submitting to the leaderboard diff --git a/configs/data-eng-bench-duckdb.codex.yaml b/configs/data-eng-bench-duckdb.codex.yaml index 2158bde8..20bd8d58 100644 --- a/configs/data-eng-bench-duckdb.codex.yaml +++ b/configs/data-eng-bench-duckdb.codex.yaml @@ -23,3 +23,4 @@ agents: model_name: openai/gpt-5.6 kwargs: reasoning_effort: high + web_search: disabled # benchmark integrity: no web solution lookup during agent.run() diff --git a/configs/data-eng-bench-duckdb.cortex-code.yaml b/configs/data-eng-bench-duckdb.cortex-code.yaml index e049b12f..6e8faa84 100644 --- a/configs/data-eng-bench-duckdb.cortex-code.yaml +++ b/configs/data-eng-bench-duckdb.cortex-code.yaml @@ -27,6 +27,13 @@ agents: model_name: anthropic/claude-opus-4-8 kwargs: reasoning_effort: high + # benchmark integrity: Cortex Code keeps web_search / web_fetch available in + # every agent mode, including code mode. Requires Harbor >= 0.22.0, which + # is where harbor-framework/harbor#2787 (adding disallowed_tools to the + # cortex-code agent) shipped. On an older Harbor, an unrecognised kwarg is + # dropped silently rather than rejected -- confirm this took effect with + # `harbor run --print-config` after upgrading. + disallowed_tools: web_search web_fetch # Portable fallback if your Harbor build has no native `cortex-code` agent: # - name: terminus-2 diff --git a/configs/data-eng-bench-snowflake.codex.yaml b/configs/data-eng-bench-snowflake.codex.yaml index bef11912..6c29e8c9 100644 --- a/configs/data-eng-bench-snowflake.codex.yaml +++ b/configs/data-eng-bench-snowflake.codex.yaml @@ -43,3 +43,4 @@ agents: model_name: openai/gpt-5.6 kwargs: reasoning_effort: high + web_search: disabled # benchmark integrity: no web solution lookup during agent.run() diff --git a/configs/data-eng-bench-snowflake.cortex-code.yaml b/configs/data-eng-bench-snowflake.cortex-code.yaml index c4c3b33d..5f8fff5b 100644 --- a/configs/data-eng-bench-snowflake.cortex-code.yaml +++ b/configs/data-eng-bench-snowflake.cortex-code.yaml @@ -43,6 +43,13 @@ agents: model_name: anthropic/claude-opus-4-8 kwargs: reasoning_effort: high + # benchmark integrity: Cortex Code keeps web_search / web_fetch available in + # every agent mode, including code mode. Requires Harbor >= 0.22.0, which + # is where harbor-framework/harbor#2787 (adding disallowed_tools to the + # cortex-code agent) shipped. On an older Harbor, an unrecognised kwarg is + # dropped silently rather than rejected -- confirm this took effect with + # `harbor run --print-config` after upgrading. + disallowed_tools: web_search web_fetch # Portable fallback if your Harbor build has no native `cortex-code` agent: # - name: terminus-2 diff --git a/docs/cortex-code.md b/docs/cortex-code.md index d46b1d1b..614ce27a 100644 --- a/docs/cortex-code.md +++ b/docs/cortex-code.md @@ -7,8 +7,8 @@ gotchas that hit first-time runs regardless of agent. For the base setup the main [README](../README.md) — this page only covers what's different or missing there. -Verified against Harbor 0.21.0 (latest published is 0.22.0 as of writing) and -Cortex Code v1.1.66. Re-check the commands below if either has moved on since. +Verified against Harbor 0.22.0 and Cortex Code v1.1.66. Re-check the commands +below if either has moved on since. --- @@ -21,6 +21,7 @@ Harbor. These account for nearly every first-run failure. | Trap | Instead | |---|---| | **Harbor 0.20.x has no `cortex-code` agent.** It landed after the 0.20.0 release. | Require **harbor >= 0.21.0**. Check `harbor --version`. | +| **`disallowed_tools` / `cli_mode` are silently ignored below 0.22.0.** The agent exists on 0.21.0, but those two kwargs landed later ([harbor#2787](https://github.com/harbor-framework/harbor/pull/2787), [#2774](https://github.com/harbor-framework/harbor/pull/2774)) and an unrecognized kwarg is dropped, not rejected — see [Keeping results honest](#keeping-results-honest). | Require **harbor >= 0.22.0** if you use either. | | **`--task-name` does not exist.** Harbor exits with "No such option". | **`-i`** / `--include-task-name` (accepts globs). | | **`--env DB_TYPE=duckdb` fails.** `--env`/`-e` selects the environment *type* (docker, modal, …), not variables. | Use a shipped **`--config`**, which sets `environment.env` for you. | | **Cortex Code needs Snowflake credentials even on DuckDB.** The DuckDB variant is hermetic for the *data*, but the agent still authenticates to Snowflake to reach its models. Without them the run dies immediately. | **`--ae SNOWFLAKE_ACCOUNT=… --ae SNOWFLAKE_USER=… --ae SNOWFLAKE_PAT=…`** | @@ -93,8 +94,8 @@ and `web_fetch` available in **every** agent mode, including code mode, so they have to be switched off explicitly — unlike `claude-code`, whose bundled config already disables web tools via `disallowed_tools`. -Harbor's `cortex-code` agent supports this via `disallowed_tools`, but **only -on Harbor main, not the latest release**: +Harbor's `cortex-code` agent supports this via `disallowed_tools`, added in +Harbor 0.22.0: ```yaml agent: @@ -103,12 +104,10 @@ agent: disallowed_tools: web_search web_fetch ``` -It merged after Harbor 0.21.0, so `uv tool install harbor` does not have it -yet. Either install from main (`uv tool install -git+https://github.com/harbor-framework/harbor`) or rely on the network -allowlist below. Setting `disallowed_tools` on a release without it is worse -than useless: Harbor drops an unrecognized kwarg **silently**, so the config -reads as protected while web access stays on. +Require **harbor >= 0.22.0** for this to take effect. On an older release, +Harbor drops an unrecognized kwarg **silently**, so the config reads as +protected while web access stays on — if you're stuck below 0.22.0, rely on +the network allowlist below instead. Space-separated, not comma-separated, if you pass this as a string on the raw CLI — the flag parses as an array and nothing splits on commas there, so @@ -207,18 +206,17 @@ Cortex Code's `--mode code` narrows the agent to a file-and-shell tool surface, dropping the Snowflake data suite, teams, cron, goals, and MCP tools. On a dbt benchmark that is a meaningfully different configuration to measure. -**Merged, not released.** Harbor's `cortex-code` agent gained `cli_mode` after -0.21.0: +**Requires Harbor >= 0.22.0.** Harbor's `cortex-code` agent gained `cli_mode` +in that release: ```yaml kwargs: cli_mode: code ``` -Same caveat as `disallowed_tools` above — install Harbor from main to get it; -on a release without it, it's silently ignored. The CLI flag itself works in -headless runs, so Option B above can use `--mode code` today regardless of -Harbor version. +Same version floor as `disallowed_tools` above — on an older release it's +silently ignored. The CLI flag itself works in headless runs, so Option B +above can use `--mode code` today regardless of Harbor version. Note code mode does **not** remove `web_search` / `web_fetch` — it drops the Snowflake data suite, teams, cron, goals, and MCP tools, but keeps both web