From b84b49adba48995be297c1f6b38705ccac5398a0 Mon Sep 17 00:00:00 2001 From: cezara98t Date: Tue, 4 Aug 2026 15:21:33 +0300 Subject: [PATCH] fix(tests): repoint the model-options vacuity guard at the Python mock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `list_model_options` (#2361, Jul 30) asserts the sh mock's redirect `>> "$(dirname "$0")/calls.log"` to prove its `projects list-models` negative guard is not vacuous. #2375 (Aug 3) rewrote mocks/uip in Python, deleting that line, so the criterion could no longer pass — the task scored 10.0/11.0 = 0.91 in the 2026-08-04 nightly with every substantive check green. Assert the Python mock's `FLAT_LOG` assignment instead: calls.log is the sink the guard above grades, so this names the coupling directly. The mock README already documents this trap ("It broke once already"); #2375 updated the README but missed this task, authored four days earlier. Comment now points at the mock so the next refactor catches it. Not a blanket rename: mock_template_ambiguous and mock_template_taxonomy are still sh mocks and legitimately keep the old redirect. Both are overlays listed second, so they shadow the Python mock in the two tasks that use them. Co-Authored-By: Claude Opus 5 --- tests/tasks/uipath-ixp/smoke/list_model_options.yaml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml index 4db4bfaa83..a85e54117a 100644 --- a/tests/tasks/uipath-ixp/smoke/list_model_options.yaml +++ b/tests/tasks/uipath-ixp/smoke/list_model_options.yaml @@ -83,10 +83,14 @@ success_criteria: pass_threshold: 1.0 # Without this, re-pointing the mock's sink leaves the seeded log clean and the - # guard above passes vacuously. + # guard above passes vacuously. Asserts the FLAT_LOG assignment because + # calls.log is the sink graded above. Coupled to mock source text (the + # documented weak fallback — no `uip` call is guaranteed in a correct run, so + # there is no positive control): keep in sync with + # ../_shared/mock_template/mocks/uip when that mock is refactored. - type: file_contains description: "Mock still records invocations to the graded calls.log (guard is not vacuous)" path: "mocks/uip" - includes: ['>> "$(dirname "$0")/calls.log"'] + includes: ['FLAT_LOG = os.path.join(MOCK_DIR, "calls.log")'] weight: 1.0 pass_threshold: 1.0