From 7d9c04cff8a8a0acee20f234257bd5e1931a9930 Mon Sep 17 00:00:00 2001 From: Daniel Date: Tue, 18 Aug 2026 20:31:36 -0700 Subject: [PATCH 1/4] Disable web tools in the codex configs; document the cortex-code gap The tasks and their reference solutions are public, so an agent that can reach the web can look up the answer instead of doing the dbt work. The bundled claude-code configs already disable web tools, but the codex configs did not, and the README's "disable the equivalent browsing tools for other agents" is advice that cannot currently be followed for cortex-code. - codex configs: add `web_search: disabled`. Harbor's codex agent already supports this as a CLI flag (`-c web_search=disabled`), so it takes effect immediately. Verified with `harbor run --print-config` against Harbor 0.21.0. - cortex-code configs: Cortex Code keeps `web_search` and `web_fetch` available in EVERY agent mode, including code mode, and Harbor's cortex-code agent has no option to switch them off. The setting is included but commented out, pointing at harbor-framework/harbor#2787, which adds it. It is deliberately not left active: Harbor drops an unrecognised kwarg silently rather than rejecting it, so an active line would read as protection while changing nothing. - README: replace the per-agent advice with a table of what each bundled config actually does, state the cortex-code gap plainly, and promote the `--allow-agent-host` network allowlist as the stricter control that does not depend on agent support. Also warn that an unrecognised kwarg is dropped silently, so a new agent's setting should be checked rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 34 +++++++++++++++---- configs/data-eng-bench-duckdb.codex.yaml | 1 + .../data-eng-bench-duckdb.cortex-code.yaml | 5 +++ configs/data-eng-bench-snowflake.codex.yaml | 1 + .../data-eng-bench-snowflake.cortex-code.yaml | 5 +++ 5 files changed, 40 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 599c6c9c..64718868 100644 --- a/README.md +++ b/README.md @@ -153,12 +153,34 @@ on a free-tier account; use the fast subset for cost-bounded runs. ## Benchmark integrity -Agents should not be able to look up reference solutions during a run. The -bundled `claude-code` configs disable web tools -(`disallowed_tools: WebSearch,WebFetch`); disable the equivalent browsing tools -for other agents. For stricter isolation, run under Harbor's network allowlist -with `--allow-agent-host`: the DuckDB variant needs only your model API host, -and the Snowflake variant also needs `.snowflakecomputing.com`. +Agents should not be able to look up reference solutions during a run. This +matters more now that the tasks and their reference solutions are public. + +The bundled configs disable web tools where the agent supports it: + +| Config | Web tools | +|---|---| +| `claude-code` | off — `disallowed_tools: WebSearch,WebFetch` | +| `codex` | off — `web_search: disabled` | +| `cortex-code` | **on** — see below | +| `terminus-2` (the commented fallback) | none exist | + +Cortex Code keeps `web_search` and `web_fetch` available in every agent mode, +including code mode, and Harbor's `cortex-code` agent has no option to switch +them off ([harbor#2787](https://github.com/harbor-framework/harbor/pull/2787) +adds one). Until that ships, isolate `cortex-code` runs at the network layer. + +For any agent, Harbor's network allowlist is the stricter control, and the only +one that does not depend on agent support: + +```bash +harbor run ... --allow-agent-host +``` + +The DuckDB variant needs only your model API host; the Snowflake variant also +needs `.snowflakecomputing.com`. If you add a config for another agent, +disable its browsing tools too — and check that the setting took effect, since +an unrecognised kwarg is dropped silently rather than rejected. ## Submitting to the leaderboard diff --git a/configs/data-eng-bench-duckdb.codex.yaml b/configs/data-eng-bench-duckdb.codex.yaml index 2158bde8..20bd8d58 100644 --- a/configs/data-eng-bench-duckdb.codex.yaml +++ b/configs/data-eng-bench-duckdb.codex.yaml @@ -23,3 +23,4 @@ agents: model_name: openai/gpt-5.6 kwargs: reasoning_effort: high + web_search: disabled # benchmark integrity: no web solution lookup during agent.run() diff --git a/configs/data-eng-bench-duckdb.cortex-code.yaml b/configs/data-eng-bench-duckdb.cortex-code.yaml index e049b12f..f764b177 100644 --- a/configs/data-eng-bench-duckdb.cortex-code.yaml +++ b/configs/data-eng-bench-duckdb.cortex-code.yaml @@ -27,6 +27,11 @@ agents: model_name: anthropic/claude-opus-4-8 kwargs: reasoning_effort: high + # benchmark integrity: Cortex Code keeps web_search / web_fetch available in + # every agent mode, and Harbor's cortex-code agent has no way to switch them + # off yet (harbor-framework/harbor#2787). Uncomment once that ships; until + # then use --allow-agent-host to keep the agent off the open internet. + # disallowed_tools: web_search web_fetch # Portable fallback if your Harbor build has no native `cortex-code` agent: # - name: terminus-2 diff --git a/configs/data-eng-bench-snowflake.codex.yaml b/configs/data-eng-bench-snowflake.codex.yaml index 7f0eb663..bcd8185b 100644 --- a/configs/data-eng-bench-snowflake.codex.yaml +++ b/configs/data-eng-bench-snowflake.codex.yaml @@ -40,3 +40,4 @@ agents: model_name: openai/gpt-5.6 kwargs: reasoning_effort: high + web_search: disabled # benchmark integrity: no web solution lookup during agent.run() diff --git a/configs/data-eng-bench-snowflake.cortex-code.yaml b/configs/data-eng-bench-snowflake.cortex-code.yaml index 02f81fba..f2e056a3 100644 --- a/configs/data-eng-bench-snowflake.cortex-code.yaml +++ b/configs/data-eng-bench-snowflake.cortex-code.yaml @@ -40,6 +40,11 @@ agents: model_name: anthropic/claude-opus-4-8 kwargs: reasoning_effort: high + # benchmark integrity: Cortex Code keeps web_search / web_fetch available in + # every agent mode, and Harbor's cortex-code agent has no way to switch them + # off yet (harbor-framework/harbor#2787). Uncomment once that ships; until + # then use --allow-agent-host to keep the agent off the open internet. + # disallowed_tools: web_search web_fetch # Portable fallback if your Harbor build has no native `cortex-code` agent: # - name: terminus-2 From 337e5e2afea571255559ded843055b61b90221a9 Mon Sep 17 00:00:00 2001 From: Daniel Date: Tue, 18 Aug 2026 21:44:44 -0700 Subject: [PATCH 2/4] Clarify that harbor#2787 is merged but not yet released The previous comment said Harbor had 'no way to switch them off yet', which is now stale: #2787 is merged. It is not in a release though -- v0.21.0 does not carry it -- and Harbor drops an unrecognised kwarg silently, so uncommenting before the next release would read as protection while changing nothing. Co-Authored-By: Claude Opus 5 (1M context) --- configs/data-eng-bench-duckdb.cortex-code.yaml | 7 +++++-- configs/data-eng-bench-snowflake.cortex-code.yaml | 7 +++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/configs/data-eng-bench-duckdb.cortex-code.yaml b/configs/data-eng-bench-duckdb.cortex-code.yaml index f764b177..eff3b84c 100644 --- a/configs/data-eng-bench-duckdb.cortex-code.yaml +++ b/configs/data-eng-bench-duckdb.cortex-code.yaml @@ -28,8 +28,11 @@ agents: kwargs: reasoning_effort: high # benchmark integrity: Cortex Code keeps web_search / web_fetch available in - # every agent mode, and Harbor's cortex-code agent has no way to switch them - # off yet (harbor-framework/harbor#2787). Uncomment once that ships; until + # every agent mode, including code mode. Harbor's cortex-code agent gained + # disallowed_tools in harbor-framework/harbor#2787, which is MERGED but not + # in a release yet -- v0.21.0 does not have it, and Harbor drops an unknown + # kwarg silently, so uncommenting before the next release would read as + # protection while changing nothing. Uncomment once a release ships; until # then use --allow-agent-host to keep the agent off the open internet. # disallowed_tools: web_search web_fetch diff --git a/configs/data-eng-bench-snowflake.cortex-code.yaml b/configs/data-eng-bench-snowflake.cortex-code.yaml index f2e056a3..c794ade6 100644 --- a/configs/data-eng-bench-snowflake.cortex-code.yaml +++ b/configs/data-eng-bench-snowflake.cortex-code.yaml @@ -41,8 +41,11 @@ agents: kwargs: reasoning_effort: high # benchmark integrity: Cortex Code keeps web_search / web_fetch available in - # every agent mode, and Harbor's cortex-code agent has no way to switch them - # off yet (harbor-framework/harbor#2787). Uncomment once that ships; until + # every agent mode, including code mode. Harbor's cortex-code agent gained + # disallowed_tools in harbor-framework/harbor#2787, which is MERGED but not + # in a release yet -- v0.21.0 does not have it, and Harbor drops an unknown + # kwarg silently, so uncommenting before the next release would read as + # protection while changing nothing. Uncomment once a release ships; until # then use --allow-agent-host to keep the agent off the open internet. # disallowed_tools: web_search web_fetch From 881b794f69ab6a24517e160152101413c9d13dba Mon Sep 17 00:00:00 2001 From: Daniel Date: Wed, 2 Sep 2026 14:59:49 -0700 Subject: [PATCH 3/4] Uncomment cortex-code disallowed_tools now that Harbor 0.22.0 ships it harbor-framework/harbor#2787 was merged but unreleased when this PR was opened, so the cortex-code disallowed_tools line was left commented out rather than silently no-op on Harbor 0.21.0. Harbor 0.22.0 shipped on 2026-08-22 and includes that PR. Verified with `harbor run --print-config` against an installed Harbor 0.22.0: disallowed_tools: "web_search web_fetch" is present in the resolved config kwargs, so it takes effect rather than being dropped. Co-Authored-By: Claude Sonnet 5 --- README.md | 10 ++++++---- configs/data-eng-bench-duckdb.cortex-code.yaml | 13 ++++++------- configs/data-eng-bench-snowflake.cortex-code.yaml | 13 ++++++------- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 64718868..935326b0 100644 --- a/README.md +++ b/README.md @@ -162,13 +162,15 @@ The bundled configs disable web tools where the agent supports it: |---|---| | `claude-code` | off — `disallowed_tools: WebSearch,WebFetch` | | `codex` | off — `web_search: disabled` | -| `cortex-code` | **on** — see below | +| `cortex-code` | off — `disallowed_tools: web_search web_fetch` (requires Harbor >= 0.22.0) | | `terminus-2` (the commented fallback) | none exist | Cortex Code keeps `web_search` and `web_fetch` available in every agent mode, -including code mode, and Harbor's `cortex-code` agent has no option to switch -them off ([harbor#2787](https://github.com/harbor-framework/harbor/pull/2787) -adds one). Until that ships, isolate `cortex-code` runs at the network layer. +including code mode. Harbor's `cortex-code` agent only gained the option to +switch them off in [harbor#2787](https://github.com/harbor-framework/harbor/pull/2787), +which shipped in Harbor 0.22.0. On an older Harbor the kwarg is dropped +silently rather than rejected, so if you're not on 0.22.0+, isolate +`cortex-code` runs at the network layer instead (below). For any agent, Harbor's network allowlist is the stricter control, and the only one that does not depend on agent support: diff --git a/configs/data-eng-bench-duckdb.cortex-code.yaml b/configs/data-eng-bench-duckdb.cortex-code.yaml index eff3b84c..6e8faa84 100644 --- a/configs/data-eng-bench-duckdb.cortex-code.yaml +++ b/configs/data-eng-bench-duckdb.cortex-code.yaml @@ -28,13 +28,12 @@ agents: kwargs: reasoning_effort: high # benchmark integrity: Cortex Code keeps web_search / web_fetch available in - # every agent mode, including code mode. Harbor's cortex-code agent gained - # disallowed_tools in harbor-framework/harbor#2787, which is MERGED but not - # in a release yet -- v0.21.0 does not have it, and Harbor drops an unknown - # kwarg silently, so uncommenting before the next release would read as - # protection while changing nothing. Uncomment once a release ships; until - # then use --allow-agent-host to keep the agent off the open internet. - # disallowed_tools: web_search web_fetch + # every agent mode, including code mode. Requires Harbor >= 0.22.0, which + # is where harbor-framework/harbor#2787 (adding disallowed_tools to the + # cortex-code agent) shipped. On an older Harbor, an unrecognised kwarg is + # dropped silently rather than rejected -- confirm this took effect with + # `harbor run --print-config` after upgrading. + disallowed_tools: web_search web_fetch # Portable fallback if your Harbor build has no native `cortex-code` agent: # - name: terminus-2 diff --git a/configs/data-eng-bench-snowflake.cortex-code.yaml b/configs/data-eng-bench-snowflake.cortex-code.yaml index c794ade6..0d68f71d 100644 --- a/configs/data-eng-bench-snowflake.cortex-code.yaml +++ b/configs/data-eng-bench-snowflake.cortex-code.yaml @@ -41,13 +41,12 @@ agents: kwargs: reasoning_effort: high # benchmark integrity: Cortex Code keeps web_search / web_fetch available in - # every agent mode, including code mode. Harbor's cortex-code agent gained - # disallowed_tools in harbor-framework/harbor#2787, which is MERGED but not - # in a release yet -- v0.21.0 does not have it, and Harbor drops an unknown - # kwarg silently, so uncommenting before the next release would read as - # protection while changing nothing. Uncomment once a release ships; until - # then use --allow-agent-host to keep the agent off the open internet. - # disallowed_tools: web_search web_fetch + # every agent mode, including code mode. Requires Harbor >= 0.22.0, which + # is where harbor-framework/harbor#2787 (adding disallowed_tools to the + # cortex-code agent) shipped. On an older Harbor, an unrecognised kwarg is + # dropped silently rather than rejected -- confirm this took effect with + # `harbor run --print-config` after upgrading. + disallowed_tools: web_search web_fetch # Portable fallback if your Harbor build has no native `cortex-code` agent: # - name: terminus-2 From ea4fb982ec4994653d8050527960058d52347280 Mon Sep 17 00:00:00 2001 From: Daniel Date: Wed, 2 Sep 2026 15:04:12 -0700 Subject: [PATCH 4/4] Update docs/cortex-code.md: disallowed_tools and cli_mode ship in Harbor 0.22.0 Both were "merged, not released" when this doc was written. Harbor 0.22.0 (released 2026-08-22) includes harbor-framework/harbor#2787 (disallowed_tools) and #2774 (cli_mode) -- verified locally with `harbor --version` (0.22.0) and `harbor run --print-config`, which shows both kwargs present in the resolved config rather than silently dropped. Co-Authored-By: Claude Sonnet 5 --- docs/cortex-code.md | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/docs/cortex-code.md b/docs/cortex-code.md index d46b1d1b..614ce27a 100644 --- a/docs/cortex-code.md +++ b/docs/cortex-code.md @@ -7,8 +7,8 @@ gotchas that hit first-time runs regardless of agent. For the base setup the main [README](../README.md) — this page only covers what's different or missing there. -Verified against Harbor 0.21.0 (latest published is 0.22.0 as of writing) and -Cortex Code v1.1.66. Re-check the commands below if either has moved on since. +Verified against Harbor 0.22.0 and Cortex Code v1.1.66. Re-check the commands +below if either has moved on since. --- @@ -21,6 +21,7 @@ Harbor. These account for nearly every first-run failure. | Trap | Instead | |---|---| | **Harbor 0.20.x has no `cortex-code` agent.** It landed after the 0.20.0 release. | Require **harbor >= 0.21.0**. Check `harbor --version`. | +| **`disallowed_tools` / `cli_mode` are silently ignored below 0.22.0.** The agent exists on 0.21.0, but those two kwargs landed later ([harbor#2787](https://github.com/harbor-framework/harbor/pull/2787), [#2774](https://github.com/harbor-framework/harbor/pull/2774)) and an unrecognized kwarg is dropped, not rejected — see [Keeping results honest](#keeping-results-honest). | Require **harbor >= 0.22.0** if you use either. | | **`--task-name` does not exist.** Harbor exits with "No such option". | **`-i`** / `--include-task-name` (accepts globs). | | **`--env DB_TYPE=duckdb` fails.** `--env`/`-e` selects the environment *type* (docker, modal, …), not variables. | Use a shipped **`--config`**, which sets `environment.env` for you. | | **Cortex Code needs Snowflake credentials even on DuckDB.** The DuckDB variant is hermetic for the *data*, but the agent still authenticates to Snowflake to reach its models. Without them the run dies immediately. | **`--ae SNOWFLAKE_ACCOUNT=… --ae SNOWFLAKE_USER=… --ae SNOWFLAKE_PAT=…`** | @@ -93,8 +94,8 @@ and `web_fetch` available in **every** agent mode, including code mode, so they have to be switched off explicitly — unlike `claude-code`, whose bundled config already disables web tools via `disallowed_tools`. -Harbor's `cortex-code` agent supports this via `disallowed_tools`, but **only -on Harbor main, not the latest release**: +Harbor's `cortex-code` agent supports this via `disallowed_tools`, added in +Harbor 0.22.0: ```yaml agent: @@ -103,12 +104,10 @@ agent: disallowed_tools: web_search web_fetch ``` -It merged after Harbor 0.21.0, so `uv tool install harbor` does not have it -yet. Either install from main (`uv tool install -git+https://github.com/harbor-framework/harbor`) or rely on the network -allowlist below. Setting `disallowed_tools` on a release without it is worse -than useless: Harbor drops an unrecognized kwarg **silently**, so the config -reads as protected while web access stays on. +Require **harbor >= 0.22.0** for this to take effect. On an older release, +Harbor drops an unrecognized kwarg **silently**, so the config reads as +protected while web access stays on — if you're stuck below 0.22.0, rely on +the network allowlist below instead. Space-separated, not comma-separated, if you pass this as a string on the raw CLI — the flag parses as an array and nothing splits on commas there, so @@ -207,18 +206,17 @@ Cortex Code's `--mode code` narrows the agent to a file-and-shell tool surface, dropping the Snowflake data suite, teams, cron, goals, and MCP tools. On a dbt benchmark that is a meaningfully different configuration to measure. -**Merged, not released.** Harbor's `cortex-code` agent gained `cli_mode` after -0.21.0: +**Requires Harbor >= 0.22.0.** Harbor's `cortex-code` agent gained `cli_mode` +in that release: ```yaml kwargs: cli_mode: code ``` -Same caveat as `disallowed_tools` above — install Harbor from main to get it; -on a release without it, it's silently ignored. The CLI flag itself works in -headless runs, so Option B above can use `--mode code` today regardless of -Harbor version. +Same version floor as `disallowed_tools` above — on an older release it's +silently ignored. The CLI flag itself works in headless runs, so Option B +above can use `--mode code` today regardless of Harbor version. Note code mode does **not** remove `web_search` / `web_fetch` — it drops the Snowflake data suite, teams, cron, goals, and MCP tools, but keeps both web