From 23f7839ebb897e45778c8c9650b18d8b0effeb79 Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Tue, 11 Aug 2026 14:56:03 +0300
Subject: [PATCH 1/8] test(ixp): grade CLI calls structurally via cli_called
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Regex over the flat call log matched a rendered command string, so every
assertion mixed the verb, the positionals and the flags into one pattern whose
failure modes were invisible: a `must_match: false` guard passes whenever the
pattern fails to match, for ANY reason, including a pattern that was subtly
wrong. cli_called (coder-eval 0.9.5) matches per-facet against the recorded
argv, so verb, positionals and flags fail independently and legibly.

Migrates four smoke tasks and closes two guard holes found while porting:

- negative_guards Task 4 (`--corrections` on a boolean): the regex required the
  literal long spelling, so `-c` slipped past a guard weighted 3.0. Now declared
  `aliases: [c]`.
- negative_guards Task 5 (`mark-missing` on a predicted field): the regex's
  broad `.*f-200` did catch `-f`; a naive facet port would have narrowed it and
  reopened the hole. Declared `aliases: [f]` to keep parity.

list_model_options asserted the sh redirect `>> "$(dirname "$0")/calls.log"`,
a string the mock's Python rewrite deleted — the harness-integrity check has
been failing since. Repointed at the JSON sink, and its negative guard moved to
cli_called, which errors on an absent log instead of reading a deleted sink as
"the forbidden verb never ran".

Verified: 21 guard/trajectory pairs replayed through the released 0.9.5 checker
agree on both correct and violating argv, including every short-alias spelling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .../uipath-ixp/smoke/list_model_options.yaml  | 29 +++---
 .../uipath-ixp/smoke/negative_guards.yaml     | 98 ++++++++++---------
 .../uipath-ixp/smoke/pick_capable_model.yaml  | 14 +--
 .../tasks/uipath-ixp/smoke/rename_field.yaml  | 25 +++--
 4 files changed, 91 insertions(+), 75 deletions(-)

diff --git a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
index 4db4bfaa83..02b374681b 100644
--- a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
+++ b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
@@ -8,8 +8,8 @@ description: >
   names none) while one that knows the distinction answers from the skill.
 
   `model_options.md` is not a self-report — it carries ground truth the agent must
-  retrieve, not a claim about work performed. The guard reads the mock's calls.log
-  rather than Bash text because a correct answer quotes `configure-model --model
+  retrieve, not a claim about work performed. The guard reads the mock's invocation
+  log rather than Bash text because a correct answer quotes `configure-model --model
   <id>` and may cite `list-models` to contrast it, which a text regex false-matches.
   A correct run makes no `uip` call, so no guaranteed log line exists for the
   positive control the mock README requires — hence its documented fallback, a
@@ -73,20 +73,25 @@ success_criteria:
     pass_threshold: 1.0
 
   # Log of ACTUAL executions: quoting the verb to contrast it is fine, running it
-  # is not. `includes:` only proves the log exists and is readable.
-  - type: file_contains
-    description: "Agent did NOT mistake the catalog for version listing: mock call log has no executed `projects list-models`"
-    path: "mocks/calls.log"
-    includes: ["# seeded by mock_template"]
-    excludes: ["uip ixp projects list-models"]
+  # is not. cli_called errors out when the log is absent, so a deleted sink can no
+  # longer read as "the forbidden verb never ran".
+  - type: cli_called
+    description: "Agent did NOT mistake the catalog for version listing: no executed `projects list-models`"
+    log: "mocks/calls.jsonl"
+    verb: "ixp projects list-models"
+    min_count: 0
+    max_count: 0
     weight: 3.0
     pass_threshold: 1.0
 
-  # Without this, re-pointing the mock's sink leaves the seeded log clean and the
-  # guard above passes vacuously.
+  # A present-but-unwritten log still passes the guard above, so assert the sink
+  # in the mock source. Both strings are needed: the constant names the file the
+  # guard reads, the open() proves invocations are appended to it.
   - type: file_contains
-    description: "Mock still records invocations to the graded calls.log (guard is not vacuous)"
+    description: "Mock still records invocations to the graded calls.jsonl (guard is not vacuous)"
     path: "mocks/uip"
-    includes: ['>> "$(dirname "$0")/calls.log"']
+    includes:
+      - 'JSON_LOG = os.path.join(MOCK_DIR, "calls.jsonl")'
+      - 'with open(JSON_LOG, "a"'
     weight: 1.0
     pass_threshold: 1.0
diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index ca7f4ac205..15188ea8d8 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -67,25 +67,25 @@ initial_prompt: |
   - Do NOT prompt the user for confirmation — this is an automated test.
 
 success_criteria:
-  - type: file_matches_regex
+  # `verb` is an ordered PREFIX, so "ixp projects" covers both `list` and `get`
+  # without the regex alternation the old pattern needed.
+  - type: cli_called
     description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+projects\s+(list|get))[^\r\n]*$
+    log: "mocks/calls.jsonl"
+    verb: "ixp projects"
     weight: 3.0
     pass_threshold: 1.0
 
-  - type: file_matches_regex
-    description: "Task 2: agent recorded verdict via uip ixp labellings (not manual extraction) — Critical Rule #11"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+labellings\s+confirm\s+["']?my_invoices-f1afa9ef-ixp["']?\s+["']?doc-abc-123["']?)[^\r\n]*$
+  - type: cli_called
+    description: "Task 2: agent recorded verdict via uip ixp labellings, not manual extraction (Critical Rule #11)"
+    log: "mocks/calls.jsonl"
+    verb: "ixp labellings confirm"
+    positional: ["my_invoices-f1afa9ef-ixp", "doc-abc-123"]
     weight: 2.0
     pass_threshold: 1.0
 
-  # Task 3: `UID` is a readonly shell variable in bash — assigning it
-  # (`UID=…`) errors out. Critical Rule #6: never name a variable `UID`;
-  # the agent must reuse the id another way (DOC_ID, a literal, etc.).
+  # Stays command_not_executed: assigning the readonly UID shell variable is a
+  # shell-level act that never reaches the CLI, so no invocation log can show it.
   - type: command_not_executed
     description: "Task 3: did NOT assign the readonly UID shell variable (Critical Rule #6)"
     tool_name: "Bash"
@@ -93,55 +93,57 @@ success_criteria:
     weight: 3.0
     pass_threshold: 1.0
 
-  - type: file_matches_regex
-    description: "Task 3: completed the task — confirmed field f-002 on the document"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+labellings\s+confirm\b[^|;&]*--fields[^|;&]*f-002)[^\r\n]*$
+  - type: cli_called
+    description: "Task 3: completed the task - confirmed field f-002 on the document"
+    log: "mocks/calls.jsonl"
+    verb: "ixp labellings confirm"
+    flags:
+      fields: {contains: "f-002", aliases: ["f"]}
     weight: 1.0
     pass_threshold: 1.0
 
-  # Task 4: --corrections is OCR-only. A boolean predicted with the
-  # wrong answer is NOT CONFIRMED — the agent must not reach for
-  # --corrections to flip the value. Critical Rule #8 (the tightened
-  # form).
-  - type: file_matches_regex
-    description: "Task 4: agent did NOT use --corrections on the boolean field (Critical Rule #8 — corrections is OCR-only)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+labellings\s+confirm\b(?=.*--corrections\s+[^|;&]*f-100).*)[^\r\n]*$
-    must_match: false
+  # Task 4: --corrections is OCR-only, never a way to flip a wrong boolean.
+  # `aliases` is load-bearing on a max_count 0 guard: without it `-c f-100`
+  # parses as an unvalued switch plus a positional, the predicate misses, and the
+  # guard passes on the exact call it forbids.
+  - type: cli_called
+    description: "Task 4: agent did NOT use --corrections on the boolean field (Critical Rule #8)"
+    log: "mocks/calls.jsonl"
+    verb: "ixp labellings confirm"
+    flags:
+      corrections: {contains: "f-100", aliases: ["c"]}
+    min_count: 0
+    max_count: 0
     weight: 3.0
     pass_threshold: 1.0
 
   # Task 5: mark-missing is only valid when IXP also predicted empty.
-  # The hint asks the agent to mark a field missing where IXP
-  # predicted a value, which is a Rule #12 violation.
-  - type: file_matches_regex
-    description: "Task 5: agent did NOT call mark-missing on a field IXP predicted a value for (Critical Rule #12)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+labellings\s+mark-missing\s+.*f-200)[^\r\n]*$
-    must_match: false
+  - type: cli_called
+    description: "Task 5: agent did NOT mark-missing a field IXP predicted a value for (Critical Rule #12)"
+    log: "mocks/calls.jsonl"
+    verb: "ixp labellings mark-missing"
+    flags:
+      fields: {contains: "f-200", aliases: ["f"]}
+    min_count: 0
+    max_count: 0
     weight: 3.0
     pass_threshold: 1.0
 
-  # Task 6: all CLI commands take the project `Name` (the -ixp slug), never the
-  # human `Title` (Critical Rule #7). The agent must pass the Name even though
-  # the hint pushes the Title.
-  - type: file_matches_regex
+  # Task 6: commands take the project Name (the -ixp slug), never the Title.
+  - type: cli_called
     description: "Task 6: did NOT pass the project Title as the project argument (Critical Rule #7)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+projects\s+get-metrics\s+["']?Quarterly\s+Reports)[^\r\n]*$
-    must_match: false
+    log: "mocks/calls.jsonl"
+    verb: "ixp projects get-metrics"
+    positional: ["Quarterly Reports"]
+    min_count: 0
+    max_count: 0
     weight: 3.0
     pass_threshold: 1.0
 
-  - type: file_matches_regex
-    description: "Task 6: used the project Name (the -ixp slug) with get-metrics (Critical Rule #7)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+projects\s+get-metrics\s+["']?quarterly_reports-7b2c-ixp)[^\r\n]*$
+  - type: cli_called
+    description: "Task 6: used the project Name with get-metrics (Critical Rule #7)"
+    log: "mocks/calls.jsonl"
+    verb: "ixp projects get-metrics"
+    positional: ["quarterly_reports-7b2c-ixp"]
     weight: 2.0
     pass_threshold: 1.0
diff --git a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
index 6ff630f990..1794a23d1b 100644
--- a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
+++ b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
@@ -27,12 +27,14 @@ initial_prompt: |
   - Do NOT prompt the user for confirmation — this is an automated test.
 
 success_criteria:
-  # Deterministic: the skill maps 100+ page docs to gemini_2_5_pro, which the
-  # 150-page prompt triggers (so gpt_4o would be wrong here).
-  - type: file_matches_regex
+  # The skill maps 100+ page docs to gemini_2_5_pro, which the 150-page prompt
+  # triggers (so gpt_4o would be wrong here).
+  - type: cli_called
     description: "Agent switched to a more capable model via `configure-model --model gemini_2_5_pro`"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+projects\s+configure-model\b(?=.*--model\s+gemini_2_5_pro\b).*)[^\r\n]*$
+    log: "mocks/calls.jsonl"
+    verb: "ixp projects configure-model"
+    positional: ["my_invoices-f1afa9ef-ixp"]
+    flags:
+      model: "gemini_2_5_pro"
     weight: 3.0
     pass_threshold: 1.0
diff --git a/tests/tasks/uipath-ixp/smoke/rename_field.yaml b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
index 8e575a1ffe..a5cbe69273 100644
--- a/tests/tasks/uipath-ixp/smoke/rename_field.yaml
+++ b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
@@ -25,19 +25,26 @@ initial_prompt: |
   - Do NOT prompt the user for confirmation — this is an automated test.
 
 success_criteria:
-  - type: file_matches_regex
+  - type: cli_called
     description: "Agent renamed field \"Amount\" to \"Total Amount\" via `fields rename` (right group)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+fields\s+rename\b(?=.*my_invoices-f1afa9ef-ixp)(?=.*--group\s+["']?Invoice Header)(?=.*--field\s+["']?Amount)(?=.*--new-name\s+["']?Total Amount).*)[^\r\n]*$
+    log: "mocks/calls.jsonl"
+    verb: "ixp fields rename"
+    positional: ["my_invoices-f1afa9ef-ixp"]
+    flags:
+      group: "Invoice Header"
+      field: "Amount"
+      new-name: "Total Amount"
     weight: 3.0
     pass_threshold: 1.0
 
-  - type: file_matches_regex
+  # A rename must not destroy the field. min_count: 0 + max_count: 0 is the
+  # must-NOT-match form; extra facets here would only give a delete more ways to
+  # slip past the guard.
+  - type: cli_called
     description: "Agent did NOT delete the field (rename must not destroy it)"
-    path: "mocks/calls.log"
-    pattern: >-
-      (?m)^(?:uip\s+ixp\s+fields\s+delete\b)[^\r\n]*$
-    must_match: false
+    log: "mocks/calls.jsonl"
+    verb: "ixp fields delete"
+    min_count: 0
+    max_count: 0
     weight: 1.0
     pass_threshold: 1.0

From f02079a9dc13aa5a469349c7d32635b36a7a8f69 Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Tue, 11 Aug 2026 16:06:55 +0300
Subject: [PATCH 2/8] test(ixp): stop the Task 1 channel check crediting
 destructive verbs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

`verb` is a literal ordered prefix with no alternation, so `verb: "ixp projects"`
cannot express the old pattern's `(list|get)` — it admits all 14 `ixp projects`
subcommands. An agent that ran `projects delete` on the project it was told to
READ therefore earned 1.0 on a weight-3.0 criterion titled "used the uip ixp
CLI". The comment claiming the prefix "covers both list and get" read as an
equivalence when the port was strictly weaker.

Adds a `max_count: 0` guard on `ixp projects delete` so the destructive case
cannot be the invocation that earns channel credit, and rewrites the comment to
state the widening instead of hiding it. `list` and `get` both still count, so
agent-visible tolerance is unchanged.

Residual, now asserted in the differential rather than assumed absent: non-delete
mutations (`update-title`, `publish`, `import-taxonomy`, ...) still earn channel
credit, as does a hallucinated subcommand. Guarding each is whack-a-mole; the
real fix is verb alternation in cli_called.

Differential: 27/27 (was 21/21) through the released 0.9.5 checker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .../uipath-ixp/smoke/negative_guards.yaml      | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index 15188ea8d8..23258d4873 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -67,8 +67,11 @@ initial_prompt: |
   - Do NOT prompt the user for confirmation — this is an automated test.
 
 success_criteria:
-  # `verb` is an ordered PREFIX, so "ixp projects" covers both `list` and `get`
-  # without the regex alternation the old pattern needed.
+  # `verb` is an ordered PREFIX and has no alternation, so this cannot express the
+  # old pattern's `(list|get)`. It is deliberately BROADER: any of the 14
+  # `ixp projects` subcommands satisfies it. That is tolerable only because the
+  # assertion is about CHANNEL (CLI, not curl) and is paired with the delete guard
+  # below — on its own it would credit `projects delete` for reading a project.
   - type: cli_called
     description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
     log: "mocks/calls.jsonl"
@@ -76,6 +79,17 @@ success_criteria:
     weight: 3.0
     pass_threshold: 1.0
 
+  # Closes the widening above: the ask is to READ metadata, so a destructive verb
+  # must never be the invocation that earns the channel credit.
+  - type: cli_called
+    description: "Task 1: did NOT destroy the project it was asked to read"
+    log: "mocks/calls.jsonl"
+    verb: "ixp projects delete"
+    min_count: 0
+    max_count: 0
+    weight: 2.0
+    pass_threshold: 1.0
+
   - type: cli_called
     description: "Task 2: agent recorded verdict via uip ixp labellings, not manual extraction (Critical Rule #11)"
     log: "mocks/calls.jsonl"

From db7252eeb28f860cf820b9e214fee9f5f7eb23cd Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Wed, 12 Aug 2026 16:02:48 +0300
Subject: [PATCH 3/8] test(ixp): drop the hardcoded subcommand count from the
 Task 1 comment
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Same rot @cezara98t flagged on the coder_eval side: "any of the 14 `ixp projects`
subcommands" is true of one CLI version, and the catalog is bot-refreshed. The point
the comment makes — that the prefix is deliberately broader than the old
`(list|get)` — does not depend on the number.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 tests/tasks/uipath-ixp/smoke/negative_guards.yaml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index 23258d4873..798a7f66e1 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -68,10 +68,10 @@ initial_prompt: |
 
 success_criteria:
   # `verb` is an ordered PREFIX and has no alternation, so this cannot express the
-  # old pattern's `(list|get)`. It is deliberately BROADER: any of the 14
-  # `ixp projects` subcommands satisfies it. That is tolerable only because the
-  # assertion is about CHANNEL (CLI, not curl) and is paired with the delete guard
-  # below — on its own it would credit `projects delete` for reading a project.
+  # old pattern's `(list|get)`. It is deliberately BROADER: any `ixp projects`
+  # subcommand satisfies it. That is tolerable only because the assertion is about
+  # CHANNEL (CLI, not curl) and is paired with the delete guard below — on its own it
+  # would credit `projects delete` for reading a project.
   - type: cli_called
     description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
     log: "mocks/calls.jsonl"

From cc22feb7261bc647c7b558ba15a9b898ff2ee7c1 Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Tue, 18 Aug 2026 12:46:44 +0300
Subject: [PATCH 4/8] test(ixp): adopt record_cli and verb_any_of from
 coder_eval main
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Drops the hand-written mock from the four cli_called smoke tasks and makes the
Task 1 channel assertion discriminate, using two features merged upstream after
this branch was written.

`sandbox.record_cli` (coder_eval 0.9.6) generates the recording shims, seeds the
log and PATH-prepends `cli_mocks/` itself, so the four tasks drop
`mock_path_dirs`, `template_sources`, and every `log:` line. The shims keep the
mock's agent-visible behavior: same stderr text, same exit 1. Two things the
hand-written mock could not do come for free — a `.cmd` twin (so these tasks are
no longer Linux-only by construction) and a framework-seeded empty log, which is
what let `list_model_options` delete its harness-integrity check rather than
repoint it. That check asserted mock source text and had been failing since
#2375; the guarantee it approximated is now the harness's.

`verb_any_of` (unreleased; coder_eval f7e9fda) restores the alternation `verb`
could not express. The interim `verb: "ixp projects"` credited ANY subcommand for
"fetch the project metadata" — replaying it shows `delete`, `configure-model`,
`get-metrics`, `get-taxonomy` and `list-models` all scoring 1.0. The paired
delete guard stays, but it is no longer patching that hole.

`_shared/mock_template` is untouched; 41 tasks still use it. Its README now says
which tasks don't and why its `calls.jsonl` sink has no consumer today.

Verification, against the real code at coder_eval f7e9fda (editable install, not
a wheel): 47 guard x trajectory pairs agree, replayed through actual generated
shims that were executed to produce the log. All 13 asserted verb spellings exist
in `assets/uip-catalog-snapshot.json`.

The pin now reads 0.9.7, which is not on PyPI yet — `verb_any_of` is merged but
unreleased, so CI cannot install a grader that understands it until that version
(or an rc of it) is published.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 tests/.coder-eval-version                     |  2 +-
 .../_shared/mock_template/README.md           | 16 +++++++-
 .../uipath-ixp/smoke/list_model_options.yaml  | 38 ++++++++-----------
 .../uipath-ixp/smoke/negative_guards.yaml     | 33 +++++++---------
 .../uipath-ixp/smoke/pick_capable_model.yaml  | 11 ++++--
 .../tasks/uipath-ixp/smoke/rename_field.yaml  | 12 +++---
 6 files changed, 58 insertions(+), 54 deletions(-)

diff --git a/tests/.coder-eval-version b/tests/.coder-eval-version
index b0bb878545..c81aa44afb 100644
--- a/tests/.coder-eval-version
+++ b/tests/.coder-eval-version
@@ -1 +1 @@
-0.9.5
+0.9.7
diff --git a/tests/tasks/uipath-ixp/_shared/mock_template/README.md b/tests/tasks/uipath-ixp/_shared/mock_template/README.md
index a315d43bde..b60f6fc01a 100644
--- a/tests/tasks/uipath-ixp/_shared/mock_template/README.md
+++ b/tests/tasks/uipath-ixp/_shared/mock_template/README.md
@@ -2,8 +2,8 @@
 
 Shape-only smoke tasks under `../../smoke/` must not authenticate or hit a live
 tenant — the smoke harness injects a live alpha bot token, so a bare `uip ixp …`
-would otherwise reach designtime-api on alpha (404-ing on fixture ids). Every
-smoke task therefore mocks `uip` with this template:
+would otherwise reach designtime-api on alpha (404-ing on fixture ids). Most
+smoke tasks therefore mock `uip` with this template:
 
 ```yaml
 sandbox:
@@ -74,6 +74,18 @@ Prefer a real positive control wherever a correct run makes any call at all.
 Integration/e2e tasks use `live_calls_template`; its wrapper writes the same two
 sinks in the same format, then delegates unchanged to the real CLI.
 
+## Not every task uses this template
+
+Tasks graded with `cli_called` declare `sandbox.record_cli` instead, and the
+framework generates the recorders, seeds the log and PATH-prepends `cli_mocks/`
+itself — no `mock_path_dirs`, no `template_sources`, no `log:` on the criterion.
+Prefer that for a new structured task. This template remains the answer whenever
+a recorder is not enough: `file_matches_regex` over the flat `calls.log`, a read
+that must return fixture data (`mock_template_taxonomy`), or per-invocation
+responses, none of which `record_cli` serves. `calls.jsonl` here has no consumer
+today for that reason — it is the structured sink for tasks that need this
+template's other behavior too.
+
 A task whose correct path reads before it writes cannot be graded on this mock
 alone — the read fails, so there is nothing to carry into the graded write, and
 an agent that declines to invent the values correctly stops. Overlay
diff --git a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
index 02b374681b..d432e79acf 100644
--- a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
+++ b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
@@ -8,12 +8,13 @@ description: >
   names none) while one that knows the distinction answers from the skill.
 
   `model_options.md` is not a self-report — it carries ground truth the agent must
-  retrieve, not a claim about work performed. The guard reads the mock's invocation
-  log rather than Bash text because a correct answer quotes `configure-model --model
+  retrieve, not a claim about work performed. The guard reads the invocation log
+  rather than Bash text because a correct answer quotes `configure-model --model
   <id>` and may cite `list-models` to contrast it, which a text regex false-matches.
-  A correct run makes no `uip` call, so no guaranteed log line exists for the
-  positive control the mock README requires — hence its documented fallback, a
-  harness-integrity assertion on the log sink.
+  A correct run makes no `uip` call at all, so the log's integrity is what separates
+  "agent behaved" from "the sink was never written": `record_cli` seeds the log
+  empty and `cli_called` errors when it is missing, so both halves are now the
+  harness's guarantee rather than a task-level assertion.
 
   Distinct from pick_capable_model / change_model_variant: both mutate a project
   via configure-model; this enumerates the catalog and mutates nothing.
@@ -25,9 +26,13 @@ run_limits:
   expected_turns: 4
 
 sandbox:
-  mock_path_dirs: [mocks]
-  template_sources:
-    - {type: template_dir, path: ../_shared/mock_template}
+  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
+  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
+  record_cli:
+    - tool: uip
+      stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
+    - tool: curl
+      stderr: "curl (mock): network egress is disabled in this offline smoke sandbox.\n"
 
 initial_prompt: |
   List available model options for IXP extraction, and write them to
@@ -73,25 +78,12 @@ success_criteria:
     pass_threshold: 1.0
 
   # Log of ACTUAL executions: quoting the verb to contrast it is fine, running it
-  # is not. cli_called errors out when the log is absent, so a deleted sink can no
-  # longer read as "the forbidden verb never ran".
+  # is not. The generated recorder seeds the log empty, so "zero calls" and "no log
+  # at all" stay distinguishable — cli_called errors on the latter.
   - type: cli_called
     description: "Agent did NOT mistake the catalog for version listing: no executed `projects list-models`"
-    log: "mocks/calls.jsonl"
     verb: "ixp projects list-models"
     min_count: 0
     max_count: 0
     weight: 3.0
     pass_threshold: 1.0
-
-  # A present-but-unwritten log still passes the guard above, so assert the sink
-  # in the mock source. Both strings are needed: the constant names the file the
-  # guard reads, the open() proves invocations are appended to it.
-  - type: file_contains
-    description: "Mock still records invocations to the graded calls.jsonl (guard is not vacuous)"
-    path: "mocks/uip"
-    includes:
-      - 'JSON_LOG = os.path.join(MOCK_DIR, "calls.jsonl")'
-      - 'with open(JSON_LOG, "a"'
-    weight: 1.0
-    pass_threshold: 1.0
diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index 798a7f66e1..bef23fc8f9 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -14,9 +14,13 @@ run_limits:
   max_turns: 45
 
 sandbox:
-  mock_path_dirs: [mocks]
-  template_sources:
-    - {type: template_dir, path: ../_shared/mock_template}
+  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
+  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
+  record_cli:
+    - tool: uip
+      stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
+    - tool: curl
+      stderr: "curl (mock): network egress is disabled in this offline smoke sandbox.\n"
 
 initial_prompt: |
   You have six tasks. Tasks 1–5 concern IXP project
@@ -67,23 +71,20 @@ initial_prompt: |
   - Do NOT prompt the user for confirmation — this is an automated test.
 
 success_criteria:
-  # `verb` is an ordered PREFIX and has no alternation, so this cannot express the
-  # old pattern's `(list|get)`. It is deliberately BROADER: any `ixp projects`
-  # subcommand satisfies it. That is tolerable only because the assertion is about
-  # CHANNEL (CLI, not curl) and is paired with the delete guard below — on its own it
-  # would credit `projects delete` for reading a project.
+  # The two verbs that read project metadata, per the skill's CLI reference. Token
+  # matching makes this narrower than the regex it replaces, whose unanchored
+  # `(list|get)` also accepted `get-taxonomy`, `get-metrics` and `list-models` —
+  # none of which answers "fetch the project metadata".
   - type: cli_called
     description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
-    log: "mocks/calls.jsonl"
-    verb: "ixp projects"
+    verb_any_of: ["ixp projects get", "ixp projects list"]
     weight: 3.0
     pass_threshold: 1.0
 
-  # Closes the widening above: the ask is to READ metadata, so a destructive verb
-  # must never be the invocation that earns the channel credit.
+  # Independent of the channel check above: the ask is to READ metadata, so a
+  # destructive verb must not run at all.
   - type: cli_called
     description: "Task 1: did NOT destroy the project it was asked to read"
-    log: "mocks/calls.jsonl"
     verb: "ixp projects delete"
     min_count: 0
     max_count: 0
@@ -92,7 +93,6 @@ success_criteria:
 
   - type: cli_called
     description: "Task 2: agent recorded verdict via uip ixp labellings, not manual extraction (Critical Rule #11)"
-    log: "mocks/calls.jsonl"
     verb: "ixp labellings confirm"
     positional: ["my_invoices-f1afa9ef-ixp", "doc-abc-123"]
     weight: 2.0
@@ -109,7 +109,6 @@ success_criteria:
 
   - type: cli_called
     description: "Task 3: completed the task - confirmed field f-002 on the document"
-    log: "mocks/calls.jsonl"
     verb: "ixp labellings confirm"
     flags:
       fields: {contains: "f-002", aliases: ["f"]}
@@ -122,7 +121,6 @@ success_criteria:
   # guard passes on the exact call it forbids.
   - type: cli_called
     description: "Task 4: agent did NOT use --corrections on the boolean field (Critical Rule #8)"
-    log: "mocks/calls.jsonl"
     verb: "ixp labellings confirm"
     flags:
       corrections: {contains: "f-100", aliases: ["c"]}
@@ -134,7 +132,6 @@ success_criteria:
   # Task 5: mark-missing is only valid when IXP also predicted empty.
   - type: cli_called
     description: "Task 5: agent did NOT mark-missing a field IXP predicted a value for (Critical Rule #12)"
-    log: "mocks/calls.jsonl"
     verb: "ixp labellings mark-missing"
     flags:
       fields: {contains: "f-200", aliases: ["f"]}
@@ -146,7 +143,6 @@ success_criteria:
   # Task 6: commands take the project Name (the -ixp slug), never the Title.
   - type: cli_called
     description: "Task 6: did NOT pass the project Title as the project argument (Critical Rule #7)"
-    log: "mocks/calls.jsonl"
     verb: "ixp projects get-metrics"
     positional: ["Quarterly Reports"]
     min_count: 0
@@ -156,7 +152,6 @@ success_criteria:
 
   - type: cli_called
     description: "Task 6: used the project Name with get-metrics (Critical Rule #7)"
-    log: "mocks/calls.jsonl"
     verb: "ixp projects get-metrics"
     positional: ["quarterly_reports-7b2c-ixp"]
     weight: 2.0
diff --git a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
index 1794a23d1b..589e1eda1c 100644
--- a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
+++ b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
@@ -11,9 +11,13 @@ run_limits:
   expected_turns: 5
 
 sandbox:
-  mock_path_dirs: [mocks]
-  template_sources:
-    - {type: template_dir, path: ../_shared/mock_template}
+  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
+  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
+  record_cli:
+    - tool: uip
+      stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
+    - tool: curl
+      stderr: "curl (mock): network egress is disabled in this offline smoke sandbox.\n"
 
 initial_prompt: |
   In the IXP project my_invoices-f1afa9ef-ixp, the documents are 150-page
@@ -31,7 +35,6 @@ success_criteria:
   # triggers (so gpt_4o would be wrong here).
   - type: cli_called
     description: "Agent switched to a more capable model via `configure-model --model gemini_2_5_pro`"
-    log: "mocks/calls.jsonl"
     verb: "ixp projects configure-model"
     positional: ["my_invoices-f1afa9ef-ixp"]
     flags:
diff --git a/tests/tasks/uipath-ixp/smoke/rename_field.yaml b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
index a5cbe69273..f9077938b1 100644
--- a/tests/tasks/uipath-ixp/smoke/rename_field.yaml
+++ b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
@@ -10,9 +10,13 @@ run_limits:
   expected_turns: 5
 
 sandbox:
-  mock_path_dirs: [mocks]
-  template_sources:
-    - {type: template_dir, path: ../_shared/mock_template}
+  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
+  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
+  record_cli:
+    - tool: uip
+      stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
+    - tool: curl
+      stderr: "curl (mock): network egress is disabled in this offline smoke sandbox.\n"
 
 initial_prompt: |
   In the IXP project my_invoices-f1afa9ef-ixp, rename the field "Amount" in
@@ -27,7 +31,6 @@ initial_prompt: |
 success_criteria:
   - type: cli_called
     description: "Agent renamed field \"Amount\" to \"Total Amount\" via `fields rename` (right group)"
-    log: "mocks/calls.jsonl"
     verb: "ixp fields rename"
     positional: ["my_invoices-f1afa9ef-ixp"]
     flags:
@@ -42,7 +45,6 @@ success_criteria:
   # slip past the guard.
   - type: cli_called
     description: "Agent did NOT delete the field (rename must not destroy it)"
-    log: "mocks/calls.jsonl"
     verb: "ixp fields delete"
     min_count: 0
     max_count: 0

From dc5a0a83f2ae9467a25ce65233eea5e72efe5b03 Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Wed, 19 Aug 2026 09:33:37 +0300
Subject: [PATCH 5/8] test(deps): pin coder_eval 0.10.2
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

`verb_any_of` shipped in 0.10.0, not the 0.9.7 this branch guessed at — the line
went 0.9.6 -> 0.10.0, so 0.9.7 never existed. 0.10.2 is current.

Re-verified against the released wheel from PyPI rather than a git checkout: 4
YAMLs valid, 47 guard x trajectory pairs agree.

The bump is repo-wide, so what rides along matters. 0.10.0's breaking change
(#110, run_limits.max_turns) is scoped to the codex and antigravity harnesses and
this repo runs claude-code; 0.10.1 and 0.10.2 are release plumbing only. One
thing does touch this PR: #119 evaluates read-only criteria after a terminal
agent failure, and cli_called is not on its list, so migrated criteria lose the
post-failure diagnostic their regex predecessors had. Scoring is unaffected.
Recorded as a follow-up on coder_eval, not worked around here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 tests/.coder-eval-version | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/.coder-eval-version b/tests/.coder-eval-version
index c81aa44afb..5eef0f10e8 100644
--- a/tests/.coder-eval-version
+++ b/tests/.coder-eval-version
@@ -1 +1 @@
-0.9.7
+0.10.2

From b0023acc912ccbc7d72fbba0ad7406114f650fcc Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Wed, 19 Aug 2026 10:29:53 +0300
Subject: [PATCH 6/8] test(ixp): trim comments to what the criteria cannot say
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Drops the four-times-repeated recorder comment: it restated what `record_cli`
does, which the framework documents and the block itself shows.

Cuts migration archaeology from the rest — PR references and the before/after
regex comparison — which belongs in the PR discussion, not in a file read years
from now. What stays is per-criterion reasoning: why `aliases` is load-bearing on
a max_count 0 guard, why the UID check stays `command_not_executed`, why a
`--help` exemption was considered and left off.

Comment lines: 24 -> 17 across the four tasks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .../uipath-ixp/smoke/list_model_options.yaml  | 20 ++++++-------------
 .../uipath-ixp/smoke/negative_guards.yaml     |  9 +--------
 .../uipath-ixp/smoke/pick_capable_model.yaml  |  2 --
 .../tasks/uipath-ixp/smoke/rename_field.yaml  |  7 ++-----
 4 files changed, 9 insertions(+), 29 deletions(-)

diff --git a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
index 96397520ce..7411ba5770 100644
--- a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
+++ b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
@@ -12,11 +12,9 @@ description: >
   rather than Bash text because a correct answer quotes `configure-model --model
   <id>` and may cite `list-models` to contrast it, which a text regex false-matches.
   No `uip` call is guaranteed in a correct run, so the guard carries no positive
-  control and adds no criterion for one (#2568). It forbids the trap verb only,
-  never every invocation — correct runs DO call other verbs while orienting. What
-  a positive control was protecting against is now structural: `record_cli` seeds
-  the log and `cli_called` errors when it is missing, so a re-pointed sink cannot
-  read as "the forbidden verb never ran".
+  control and asserts none. It forbids the trap verb only, never every invocation —
+  correct runs DO call other verbs while orienting. A re-pointed sink cannot read as
+  "never ran": the log is seeded by the harness, and a missing one errors.
 
   Distinct from pick_capable_model / change_model_variant: both mutate a project
   via configure-model; this enumerates the catalog and mutates nothing.
@@ -28,8 +26,6 @@ run_limits:
   expected_turns: 4
 
 sandbox:
-  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
-  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
   record_cli:
     - tool: uip
       stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
@@ -80,13 +76,9 @@ success_criteria:
     pass_threshold: 1.0
 
   # Log of ACTUAL executions: quoting the verb to contrast it is fine, running it
-  # is not. This is the structured form #2568 asked for: the verb path matches as
-  # tokens, so no seed-line include is needed to prove the log exists — a missing
-  # log errors instead of reading as "never ran".
-  #
-  # A `--help` probe is now exemptable (`flags: {help: {absent: true}}`), left off
-  # deliberately: a max_count 0 guard wants the fewest facets that capture the
-  # forbidden act, and none has been observed. Add it if one ever is.
+  # is not. A `--help` probe would be exemptable (`flags: {help: {absent: true}}`);
+  # left off, since a max_count 0 guard wants the fewest facets and none has been
+  # observed.
   - type: cli_called
     description: "Agent did NOT mistake the catalog for version listing: no executed `projects list-models`"
     verb: "ixp projects list-models"
diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index bef23fc8f9..396b09425d 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -14,8 +14,6 @@ run_limits:
   max_turns: 45
 
 sandbox:
-  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
-  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
   record_cli:
     - tool: uip
       stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
@@ -71,18 +69,13 @@ initial_prompt: |
   - Do NOT prompt the user for confirmation — this is an automated test.
 
 success_criteria:
-  # The two verbs that read project metadata, per the skill's CLI reference. Token
-  # matching makes this narrower than the regex it replaces, whose unanchored
-  # `(list|get)` also accepted `get-taxonomy`, `get-metrics` and `list-models` —
-  # none of which answers "fetch the project metadata".
+  # The two verbs that read project metadata, per the skill's CLI reference.
   - type: cli_called
     description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
     verb_any_of: ["ixp projects get", "ixp projects list"]
     weight: 3.0
     pass_threshold: 1.0
 
-  # Independent of the channel check above: the ask is to READ metadata, so a
-  # destructive verb must not run at all.
   - type: cli_called
     description: "Task 1: did NOT destroy the project it was asked to read"
     verb: "ixp projects delete"
diff --git a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
index 589e1eda1c..0a1a7b4e5d 100644
--- a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
+++ b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
@@ -11,8 +11,6 @@ run_limits:
   expected_turns: 5
 
 sandbox:
-  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
-  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
   record_cli:
     - tool: uip
       stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
diff --git a/tests/tasks/uipath-ixp/smoke/rename_field.yaml b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
index f9077938b1..5356b52715 100644
--- a/tests/tasks/uipath-ixp/smoke/rename_field.yaml
+++ b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
@@ -10,8 +10,6 @@ run_limits:
   expected_turns: 5
 
 sandbox:
-  # Generated recorders: the sandbox shims uip/curl into cli_mocks/ and appends
-  # each invocation to cli_mocks/calls.jsonl, the log cli_called reads by default.
   record_cli:
     - tool: uip
       stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
@@ -40,9 +38,8 @@ success_criteria:
     weight: 3.0
     pass_threshold: 1.0
 
-  # A rename must not destroy the field. min_count: 0 + max_count: 0 is the
-  # must-NOT-match form; extra facets here would only give a delete more ways to
-  # slip past the guard.
+  # A rename must not destroy the field. Extra facets here would only give a
+  # delete more ways to slip past the guard.
   - type: cli_called
     description: "Agent did NOT delete the field (rename must not destroy it)"
     verb: "ixp fields delete"

From 92ad9c61ebd980bd2c980173104c0c2f7ba26bcb Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Wed, 19 Aug 2026 15:07:46 +0300
Subject: [PATCH 7/8] test(ixp): anchor every cli_called criterion to tool: uip
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Copilot caught a real regression in the port. `record_cli` shims uip AND curl into
one shared log, and `cli_called` treats an omitted `tool` as "any tool" — so the
argv facets alone decide the match. The regexes this replaced were anchored with
`^uip\s+`, and dropping that widened all 12 criteria on an axis nobody intended.

Reproduced before fixing, with curl records shaped like the uip verb: 5 positive
assertions credited a curl-only trajectory and 2 max_count 0 guards tripped on
one. The worst is Task 1, whose entire claim is "used the CLI, not curl" — it
scored 1.0 on `curl ixp projects get`, inverting the assertion it exists to make.

A real curl call is `curl https://...`, whose argv matches none of these, so no
observed run was misgraded. That makes it latent, not harmless: the guard was one
plausible agent behavior away from lying.

Fixed with `tool: "uip"` on all 12. On a negative guard this adds a facet, which
normally weakens a max_count 0 check — it does not here, because it excludes only
records that cannot be the forbidden `uip` call. Noted in the file, since that is
the distinction between this and the `--help` carve-out left off earlier.

Verified: the curl attack drops 7 fooled criteria to 0, and the 47 guard x
trajectory pairs still agree on coder-eval 0.10.2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 tests/tasks/uipath-ixp/smoke/list_model_options.yaml |  1 +
 tests/tasks/uipath-ixp/smoke/negative_guards.yaml    | 12 ++++++++++++
 tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml |  1 +
 tests/tasks/uipath-ixp/smoke/rename_field.yaml       |  2 ++
 4 files changed, 16 insertions(+)

diff --git a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
index 7411ba5770..84e3746aae 100644
--- a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
+++ b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml
@@ -82,6 +82,7 @@ success_criteria:
   - type: cli_called
     description: "Agent did NOT mistake the catalog for version listing: no executed `projects list-models`"
     verb: "ixp projects list-models"
+    tool: "uip"
     min_count: 0
     max_count: 0
     weight: 3.0
diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index 396b09425d..d13004de11 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -70,15 +70,21 @@ initial_prompt: |
 
 success_criteria:
   # The two verbs that read project metadata, per the skill's CLI reference.
+  # `tool` is required, not optional: uip and curl share one log, so without it a
+  # `curl ixp projects get` satisfies the very assertion that curl was refused.
+  # Every criterion here carries it — on a max_count 0 guard it excludes only
+  # records that cannot be the forbidden `uip` call, so it costs no coverage.
   - type: cli_called
     description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
     verb_any_of: ["ixp projects get", "ixp projects list"]
+    tool: "uip"
     weight: 3.0
     pass_threshold: 1.0
 
   - type: cli_called
     description: "Task 1: did NOT destroy the project it was asked to read"
     verb: "ixp projects delete"
+    tool: "uip"
     min_count: 0
     max_count: 0
     weight: 2.0
@@ -87,6 +93,7 @@ success_criteria:
   - type: cli_called
     description: "Task 2: agent recorded verdict via uip ixp labellings, not manual extraction (Critical Rule #11)"
     verb: "ixp labellings confirm"
+    tool: "uip"
     positional: ["my_invoices-f1afa9ef-ixp", "doc-abc-123"]
     weight: 2.0
     pass_threshold: 1.0
@@ -103,6 +110,7 @@ success_criteria:
   - type: cli_called
     description: "Task 3: completed the task - confirmed field f-002 on the document"
     verb: "ixp labellings confirm"
+    tool: "uip"
     flags:
       fields: {contains: "f-002", aliases: ["f"]}
     weight: 1.0
@@ -115,6 +123,7 @@ success_criteria:
   - type: cli_called
     description: "Task 4: agent did NOT use --corrections on the boolean field (Critical Rule #8)"
     verb: "ixp labellings confirm"
+    tool: "uip"
     flags:
       corrections: {contains: "f-100", aliases: ["c"]}
     min_count: 0
@@ -126,6 +135,7 @@ success_criteria:
   - type: cli_called
     description: "Task 5: agent did NOT mark-missing a field IXP predicted a value for (Critical Rule #12)"
     verb: "ixp labellings mark-missing"
+    tool: "uip"
     flags:
       fields: {contains: "f-200", aliases: ["f"]}
     min_count: 0
@@ -137,6 +147,7 @@ success_criteria:
   - type: cli_called
     description: "Task 6: did NOT pass the project Title as the project argument (Critical Rule #7)"
     verb: "ixp projects get-metrics"
+    tool: "uip"
     positional: ["Quarterly Reports"]
     min_count: 0
     max_count: 0
@@ -146,6 +157,7 @@ success_criteria:
   - type: cli_called
     description: "Task 6: used the project Name with get-metrics (Critical Rule #7)"
     verb: "ixp projects get-metrics"
+    tool: "uip"
     positional: ["quarterly_reports-7b2c-ixp"]
     weight: 2.0
     pass_threshold: 1.0
diff --git a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
index 0a1a7b4e5d..0feae5d255 100644
--- a/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
+++ b/tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml
@@ -34,6 +34,7 @@ success_criteria:
   - type: cli_called
     description: "Agent switched to a more capable model via `configure-model --model gemini_2_5_pro`"
     verb: "ixp projects configure-model"
+    tool: "uip"
     positional: ["my_invoices-f1afa9ef-ixp"]
     flags:
       model: "gemini_2_5_pro"
diff --git a/tests/tasks/uipath-ixp/smoke/rename_field.yaml b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
index 5356b52715..de7bbfadbc 100644
--- a/tests/tasks/uipath-ixp/smoke/rename_field.yaml
+++ b/tests/tasks/uipath-ixp/smoke/rename_field.yaml
@@ -30,6 +30,7 @@ success_criteria:
   - type: cli_called
     description: "Agent renamed field \"Amount\" to \"Total Amount\" via `fields rename` (right group)"
     verb: "ixp fields rename"
+    tool: "uip"
     positional: ["my_invoices-f1afa9ef-ixp"]
     flags:
       group: "Invoice Header"
@@ -43,6 +44,7 @@ success_criteria:
   - type: cli_called
     description: "Agent did NOT delete the field (rename must not destroy it)"
     verb: "ixp fields delete"
+    tool: "uip"
     min_count: 0
     max_count: 0
     weight: 1.0

From 8b8bebe4a91757c3e6cd1ca180b11d8b6503f25e Mon Sep 17 00:00:00 2001
From: Alexandru Jircan <alexandru.jircan@uipath.com>
Date: Wed, 19 Aug 2026 15:30:49 +0300
Subject: [PATCH 8/8] test(ixp): pin the document on Task 3's completion
 criterion
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

@cezara98t flagged two criteria labelled "Task 3". The labels are right — Tasks 1,
3 and 6 each pair a rule guard with a completion assertion — but checking it found
the completion half asserted less than its own description claimed.

It said "confirmed field f-002 on the document" while constraining only the verb
and `--fields`, naming no document. So confirming f-002 on Task 2's doc-abc-123
earned Task 3 credit, as did confirming it with no document at all, `document-id`
being optional on `confirm`. Three trajectories scored 1.0 that should have scored
0.0; Task 2's own criterion has pinned both positionals all along.

Carried over from the regex, which was equally document-blind
(`confirm\b[^|;&]*--fields[^|;&]*f-002`), so this predates the migration rather
than regressing in it.

Fixed by naming doc-def-456 in the positional facet and in the description, which
also makes the two Task 3 labels tell themselves apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 tests/tasks/uipath-ixp/smoke/negative_guards.yaml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
index d13004de11..f379a45766 100644
--- a/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
+++ b/tests/tasks/uipath-ixp/smoke/negative_guards.yaml
@@ -107,10 +107,13 @@ success_criteria:
     weight: 3.0
     pass_threshold: 1.0
 
+  # Names the document: without it, confirming f-002 on Task 2's doc-abc-123 — or on
+  # nothing at all, since document-id is optional on `confirm` — earned Task 3.
   - type: cli_called
-    description: "Task 3: completed the task - confirmed field f-002 on the document"
+    description: "Task 3: completed the task - confirmed field f-002 on doc-def-456"
     verb: "ixp labellings confirm"
     tool: "uip"
+    positional: ["my_invoices-f1afa9ef-ixp", "doc-def-456"]
     flags:
       fields: {contains: "f-002", aliases: ["f"]}
     weight: 1.0