test(uipath-maestro-flow): grade read-only artifact checks after a turn timeout

[rockymadden]

🔴 Merge precondition — three steps, in order

RunCommandCriterion.model_config is extra="forbid", so read_only is a hard
schema error on every currently released harness. All three must happen before
this merges:

1. Merge UiPath/coder_eval#197 — adds the field.
2. Cut a coder-eval release. main is at 0.12.6; feat(aops-governance): add uipath-aops-governance skill [PLT-100543] #197 is a feat, and on
   0.x python-semantic-release takes that to a patch bump, so it should land as
   0.12.7 — but take the number from the actual chore(release): commit.
3. Bump tests/.coder-eval-version (currently 0.12.4) to that release.
   It is the single source of truth for the installed wheel in
   run-coder-eval.yml and activation-gate.yml, so feat(aops-governance): add uipath-aops-governance skill [PLT-100543] #197 merging changes
   nothing here on its own.

Why step 3 is not optional, and why it fails quietly. A schema failure is not
loud. task_loader.load_task raises ValueError, and
orchestration/experiment.py:643 catches it, logs a warning, and appends to
RunSummary.skipped_tasks. The run continues and goes green. Merged on the
current pin, all 27 files drop out of every run as skipped tasks and the loss is
invisible unless someone reads skipped_tasks. That is worse than the false zeros
this PR fixes. Thanks to @nikhil for catching that the pin, not just the merge, is
the gate.

Where the bump goes: here. Repo precedent bundles the pin with the change that
needs it — #2352 (0.9.1, early-stop), #2864 (0.11.5, litellm checker route),
#2565 (0.9.5, cli_called) each moved tests/.coder-eval-version in the same PR
as the task-corpus change adopting the new capability. The number cannot be written
yet because the release does not exist. It lands as a final commit here, before
this leaves draft.

What this fixes

coder_eval refuses to evaluate any run_command criterion after a terminal agent
failure:

Not evaluated after terminal agent failure: the criterion is not a deterministic, read-only artifact check.

The whole uipath-maestro-flow suite grades via run_command, so a turn timeout
reports nothing at all about the artifacts sitting on disk. #197 adds read_only
as a per-criterion opt-in; this PR sets it on the criteria that genuinely qualify.

Scope check: the post-failure result is diagnostic. post_failure_criteria_results
is a separate field from success_criteria_results — it does not feed
calculate_weighted_score, task gating, or the suite_thresholds rollups
(reports/markdown.py aggregates success_criteria_results only). A timed-out run
still scores 0.00 and still reports ERROR. What changes is that the report now says
what the artifacts were worth, instead of silence.

The two proven false zeros

Both runs ended final_status: ERROR, weighted_score: 0.0 on a turn timeout.
Every grader exits 0 against the preserved sandbox:

$ cd <sandbox>/skill-flow-init-plain-default/
$ python3 _shared/project_profile.py --flow-name ShipmentTracker \
      --expect-profile Standard --expect-no-sentinel
exit=0
$ python3 _shared/flow_contains.py --flow-name ShipmentTracker
Found 1 .flow file(s): ShipmentTracker/ShipmentTracker/ShipmentTracker.flow
exit=0

$ cd <sandbox>/skill-flow-hitl-smoke-node-placed/
$ python3 _shared/flow_contains.py
Found 1 .flow file(s): InvoiceApproval/InvoiceApproval/InvoiceApproval.flow
exit=0
$ python3 _shared/check_simulated_hitl.py quick-form
OK: flow contains an inline HITL quick form
exit=0

Classification

Every checker was read before classifying — no filename heuristics.
71 of 102 run_command criteria marked, across 27 files.

Checker	Marked	Why
_shared/flow_contains.py (44)	✅	Discovery via flow_check._rglob_pruned + open()/re.search. No subprocess, no writes.
_shared/project_profile.py (2)	✅	rglob + json.loads(operate.json). Pure.
_shared/check_simulated_hitl.py (9)	✅	find_flow_file + json.loads, node-shape asserts only.
_shared/check_inline_agent.py (3)	✅	glob + json over agent.json. Stdlib only.
_shared/check_escalation_behavior.py (1)	✅	Reads the .flow source; the simulated twin of a debug checker, no run_debug.
smoke/check_merge_parallel_sync_flow.py (1)	✅	Graph inspection of the parsed .flow.
smoke/check_scheduled_trigger_flow.py (1)	✅	Trigger/timer inspection of the parsed .flow.
ixp/check_project_selection.py (1)	✅	Reads the .flow, matches modelName against a literal table keyed on cwd.
interactive/check_solution_select.py — flow, project, solution, existing-untouched, no-extra-solution (5)	✅	Path/JSON reads of .uipx and project.uiproj.
ixp/e2e_01 inline grep -rq … --include="*.flow" . (4)	✅	Plain grep over the emitted flows.
_shared/validate_flow.py (22)	❌	See below.
ixp/e2e_01 inline uip maestro flow validate (1)	❌	Same, unwrapped.
interactive/check_solution_select.py validate (1)	❌	Same, via subprocess.run(["uip", …, "validate"]).
_shared/check_weather_flow_simulated.py, check_dice_runs_simulated.py, check_channel_description_simulated.py (3)	❌	flow_check.run_debug → uip maestro flow debug starts a real cloud job. run_debug also calls _rotate_solution_id, which rewrites the .uipx on disk and appends to a sidecar.
interactive/customer_escalation_triage/check_customer_escalation_triage.py (1)	❌	Same run_debug path, with grader-supplied inputs.
_shared/check_ixp_handoff.py (1)	❌	Runs uip maestro flow registry pull --force (mutates the local registry cache, hits the tenant) plus uip ixp deployments list, and reads a .grader_baseline.json snapshot.
ixp/routing_listing.yaml, ixp/routing_negative.yaml (2)	❌	Pure and deterministic, but see Findings #1.

command_executed / command_not_executed / skill_triggered (25) are not
touched: they subclass LiveSuccessCriterion, not RunCommandCriterion, so
they never reach this code path and carry no read_only field. They grade the
trajectory, which is exactly what a timeout truncates. json_check (4) already
has supports_post_failure_evaluation = True at the class level.

On uip maestro flow validate — not marked

It is not a mutation: validate writes nothing server-side. It fails the
determinism leg, and that is enough.

validate refreshes the node manifest from the tenant on every invocation.
validate_flow.py's own implementation is the evidence: it budgets the call,
caps each attempt, retries once with a 5s backoff, and documents the fetch at
"8-14s typical and 60-67s on the tail". On exhaustion it returns 1 with
"The flow file itself may well be valid — this is the CLI's manifest fetch, not a
schema fault." Same artifacts, two verdicts, decided by tenant latency. A
criterion whose answer depends on a network round-trip is not the
"deterministic, read-only artifact check" the field is for.

Second reason to stay out: post-failure grading fires on runs that just blew their
time budget. Marking 22 criteria would add a tenant round-trip apiece to precisely
the runs already under load.

This is the conservative call by design — a missed opportunity costs nothing, a
wrongly-marked criterion is a correctness bug in the grader. The 49 pure checks
already recover the signal these 22 would have added.

Findings from my own adversarial pass

1. Negative-only tasks would report a false 1.0. ixp/routing_listing.yaml
   (flow_contains.py --expect-none) and ixp/routing_negative.yaml
   (--absent-regex uipath\.ixp) each have exactly one gradable criterion, and an
   untouched sandbox satisfies it. Marked, a turn timeout would record "the agent
   correctly refrained from building" for a run that died before it could — the
   same false signal this PR exists to remove, inverted. Fixed: both unmarked.
   Negative criteria that sit beside a positive one in the same task
   (ixp/e2e_02, interactive/solution_select) stay marked; the report cannot be
   misread there unless the positives pass too.
2. Would a seeded fixture satisfy a marked criterion for free? Checked every
   sandbox.template_sources and _setup under these four directories. The only
   seeded artifacts are interactive/fixtures/existing_solutions/{SolarReports,TideTracker}.uipx
   (empty Projects arrays). No .flow is ever seeded, and
   hitl/quality_04_brownfield_insert has the agent build its own base flow from
   the prompt. Every marked positive still requires agent work. No change needed.
3. Shell side effects in a marked command. Swept all 71 for ;, &&, >,
   $(, backticks, rm, mkdir, cd. Only hits are | inside quoted regex
   alternations. Clean.
4. Half-written artifacts. A truncated .flow raises JSONDecodeError and the
   checker exits non-zero with the filename. That is a correct fail, and identical
   on every run against the same bytes. Determinism holds.
5. Field name drift. Validated, not guessed — all 33 task files round-trip
   through TaskDefinition.model_validate on feat(aops-governance): add uipath-aops-governance skill [PLT-100543] #197's committed tree
   (24b2ab65), and RunCommandCriterion(read_only=True).evaluable_after_agent_failure
   is True.

Checks run

Check	Result
TaskDefinition.model_validate on all 33 yaml, against coder_eval #197 @ 24b2ab65	✅ valid=33 invalid=0 marked_run_command=71
pytest tests/tasks/uipath-maestro-flow/{_shared,smoke}	✅ 1199 passed in 5.75s
scripts/check-task-host-paths.py --base-ref origin/main (task-host-paths-gate)	✅ OK — no $TASK_DIR/$SKILLS_REPO_PATH references
scripts/check-task-driver.py tests/tasks tests/experiments (task-driver-gate)	✅ exit 0
Graders re-run by hand against both preserved sandboxes	✅ all exit 0 (output above)

lint-tasks.yml is draft-gated and advisory; it will fire when this comes out of
draft.

Diff

71 added lines, all of them read_only: true. No re-weighting, no re-wording, no
comment touched — git diff -U0 contains no other changed line.

🤖 Generated with Claude Code

[rockymadden]

CI: 🔴 1 red, and it is the block itself

Validate task schema (advisory) validates every task YAML against coder_eval@main,
which has not yet learned the field:

Error: Invalid task definition: 4 validation errors for TaskDefinition
success_criteria.1.run_command.read_only
  Extra inputs are not permitted

That is RunCommandCriterion.model_config = {"extra": "forbid"} doing its job. It
clears when UiPath/coder_eval#197
merges and this repo picks up the release carrying it. Nothing to fix here.

Everything else is green: 32 pass, 6 skipped (draft-gated), 1 fail. The gates
that actually cover these paths — No new task TASK_DIR/SKILLS_REPO_PATH references,
No task pins sandbox.driver tempdir, task/experiment gate unit tests — all pass.

Locally, the same 33 files validate clean against #197's committed tree
(24b2ab65): valid=33 invalid=0 marked_run_command=71.

[Copilot]

Copilot review overview

🟡 Changes recommended

The pinned coder_eval 0.12.4 schema rejects read_only; upstream PR #197 and a compatible version update must land first.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: None

What changed in this PR

Adds post-timeout diagnostic grading for 71 deterministic, read-only Maestro Flow artifact checks without affecting task scores.

Changes:

• Marks eligible filesystem, JSON, and static flow checks as read_only.
• Covers smoke, HITL, IxP, and interactive tasks.
• Leaves validation, debug, and tenant-dependent checks unmarked.

File	Description
smoke/​scheduled_trigger.yaml	Marks scheduled-trigger inspection read-only.
smoke/​merge_parallel_sync.yaml	Marks merge-topology inspection read-only.
smoke/​inline_agent_robust.yaml	Marks agent-sidecar and flow checks read-only.
smoke/​init_validate.yaml	Marks initialized-flow checks read-only.
smoke/​init_plain_flow_default.yaml	Marks profile and artifact checks read-only.
smoke/​init_maestro_automate.yaml	Marks Automate profile checks read-only.
ixp/​scaffold_multinode.yaml	Marks multinode IxP checks read-only.
ixp/​scaffold_minimal.yaml	Marks minimal IxP checks read-only.
ixp/​routing.yaml	Marks IxP-node inspection read-only.
ixp/​integration_handle_routing.yaml	Marks routing structure checks read-only.
ixp/​e2e_02_project_selection.yaml	Marks model-selection checks read-only.
ixp/​e2e_01_invoice_extraction_greenfield.yaml	Marks static grep checks read-only.
interactive/​solution_select.yaml	Marks solution artifact checks read-only.
interactive/​slack_channel_description_simulated/​slack_channel_description_simulated.yaml	Marks Slack flow checks read-only.
interactive/​ixp_invoice_extraction_simulated/​ixp_invoice_extraction_simulated.yaml	Marks simulated IxP checks read-only.
interactive/​hitl_schema_design_simulated/​hitl_schema_design_simulated.yaml	Marks HITL schema checks read-only.
interactive/​expense_approval_simulated/​expense_approval_simulated.yaml	Marks expense-flow checks read-only.
interactive/​customer_escalation_simulated/​customer_escalation_simulated.yaml	Marks escalation artifact checks read-only.
interactive/​cli_dice_roller_simulated/​cli_dice_roller_simulated.yaml	Marks flow-name check read-only.
interactive/​bellevue_weather_simulated/​bellevue_weather_simulated.yaml	Marks flow-name check read-only.
hitl/​smoke_03_multi_outcome_routing.yaml	Marks HITL routing checks read-only.
hitl/​smoke_02_completed_port_wired.yaml	Marks outcome-wiring checks read-only.
hitl/​smoke_01_hitl_node_placed.yaml	Marks HITL placement checks read-only.
hitl/​quality_04_brownfield_insert.yaml	Marks brownfield preservation checks read-only.
hitl/​quality_03_boolean_decision.yaml	Marks decision-output checks read-only.
hitl/​quality_02_result_downstream.yaml	Marks downstream-result checks read-only.
hitl/​quality_01_schema_design.yaml	Marks HITL schema checks read-only.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[nikhil-maryala]

Overall decision: Comment (content ready; merge-gated on coder_eval #197)

Severity tally: 1 Blocker (merge precondition) · 0 Critical · 0 Major · 0 Minor · 1 Nit.

The classification is careful, and it matches the code. Verified:

• Uniform. 71 byte-identical read_only: true lines, all on run_command criteria, at the criterion-key indent. The two after a multi-line command: still parse as a top-level true. None is on command_executed, llm_judge, skill_triggered, or json_check.
• Only pure readers are marked. I read every marked checker: flow_contains, project_profile, check_simulated_hitl, check_inline_agent, check_escalation_behavior, check_merge_parallel_sync_flow, check_scheduled_trigger_flow, check_project_selection, and check_solution_select in its read modes. None writes or spawns a subprocess, and the flow_check helpers they reach are pure.
• Correctly left unmarked: validate_flow.py and inline validate (tenant-dependent), check_solution_select validate, the three *_simulated.py checkers that call run_debug (a cloud job that rewrites the .uipx), and the registry pull --force checkers.
• Plumbing. All 134 flow YAMLs safe_load; check-cli-verbs 0 High/0 Medium; check-task-host-paths OK. All required checks are green; the only red is the advisory schema check explained inline.

Follow-ups (outside the diff, optional):

• tests/README.md § run_command doesn't document read_only, or the rule never to mark validate/debug graders.
• About 30 other flow tasks use the same pure checkers but aren't marked (connector_features/*, evaluate/*, voice/*, multi_node/billing_*, …). That's harmless, but the scope (hitl/interactive/ixp/smoke) is worth stating or tracking.

Will approve once the pin is bumped to a release containing #197.