[rockymadden]

What

Appends an informational skill_triggered criterion for uipath-maestro-flow to single_node/coded_agent/coded_agent.yaml, after the graded criteria so their indices stay put. weight: 0.0 keeps it out of the score and the gate, and pass_threshold: 0.0 satisfies test_gating_skill_telemetry_matches_the_temporary_allowlist.

Why

The 2026-10-02 codex run (gpt-5.6-terra) scored 0.375 as if the skill were wrong, but the agent never read the skill:

• It read .agents/skills/uipath/uipath-maestro-flow/SKILL.md (wrong path, extra uipath/). Then rg and find skipped the symlinked skill dirs, and it gave up after 4 calls.
• Without the skill, it wrote =variables.sentence through flow node add. Debug faulted with [400300] Unknown identifier 'variables'.

That cause is fixed upstream: UiPath/coder_eval#225 copies skills into .agents/skills/ instead of symlinking, and main picked it up with #3982 (pin 0.12.14). The label still matters for any run that never engages the skill: it reads observed='no' instead of looking like a skill defect.

Evidence

• I replayed the criterion's _SKILL_PATH_RE over that run's commands. It captures only uipath, so it labels the run no. Positive control: the same replay on the 2026-10-02 DTL run, which did read the skill, captures uipath-maestro-flow.
• Grading budget: 180 + 720 + 30 (default) = 930s, under task_timeout 2400.

Not in this PR

• The detector counts a command that mentions the path even if the command failed, so a failed cat of the right path reads yes. That is a coder_eval change.
• coder_eval skips skill_triggered after a terminal agent failure (crash, turn or task timeout), so those runs get no label.

Verification

• pytest on _shared/test_same_ground_corpus.py and _shared/test_criterion_budgets.py after merging main: 945 passed.
• No eval was run, so this PR makes no passing-run claim.

🤖 Generated with Claude Code

[github-actions]

Claude finished @rockymadden's task in 1m 43s —— View job

---

PR Review: test(maestro-flow): label skill activation on the coded-agent eval

• Read review criteria and project conventions
• Read full diff and changed file
• Read test framework docs for criterion validation
• Check for structural and convention compliance
• Post review

---

Summary

Appends an informational skill_triggered criterion (weight: 0.0, pass_threshold: 0.0) to the coded_agent.yaml eval task, so runs where the agent never reads the skill are labeled observed='no' rather than silently inflating the skill's error rate. The change is appended after the two graded criteria, preserving their indices.

Change-by-Change Review

1. tests/tasks/uipath-maestro-flow/single_node/coded_agent/coded_agent.yaml (lines 61–66)

Severity: OK

The added criterion:

  - type: skill_triggered
    description: "Advisory: agent engaged the uipath-maestro-flow skill"
    skill_name: "uipath-maestro-flow"
    expected_skill: "uipath-maestro-flow"
    weight: 0.0
    pass_threshold: 0.0

Verified:

• Field structure matches the established skill_triggered pattern used across the repo (e.g., ixp/e2e_04_build_mechanics.yaml:111, interactive/customer_escalation_triage.yaml:76). All required fields (type, skill_name, expected_skill, weight, pass_threshold) are present.
• weight: 0.0 keeps it out of the score, so existing grading is unaffected.
• pass_threshold: 0.0 means the criterion never gates — and critically, it satisfies test_gating_skill_telemetry_matches_the_temporary_allowlist in _shared/test_same_ground_corpus.py without needing an allowlist entry (that test only flags skill_triggered criteria where pass_threshold > 0).
• Grading budget holds: 180 + 720 + 30 (default skill_triggered timeout) = 930s, well under the task_timeout: 2400 minus turn_timeout: 1200 = 1200s grading ceiling.
• Index stability: appended after the two existing graded criteria at indices 0 and 1, so their positions are unchanged — no scoring regression for existing runs.
• No secrets, tokens, or personal paths in the diff.
• Scoped change: only this one task file is modified.

What's Missing

Nothing identified. The PR is a minimal, self-contained labeling addition.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No frontmatter changes
E2E Tests	OK	Criterion appended correctly with weight-0 / threshold-0 convention
Skill Body	N/A	No skill content changes
References & Assets	N/A	No reference or asset changes
Repo Hygiene	OK	No secrets, scoped change, no unrelated modifications

Issues for Manual Review

• The PR description notes this was not run and makes no passing-run claim. The criterion itself is non-gating (weight 0), so there is no correctness risk from skipping a run, but the _SKILL_PATH_RE regex behavior described in the PR body (matching on command paths) is untested against the actual coder_eval implementation. Worth a spot-check if the labeling is important for downstream dashboards.

Conclusion

Clean, minimal change. The skill_triggered criterion follows established repo conventions, does not affect scoring, and correctly avoids the telemetry allowlist gate. No issues found — approve.

[github-actions]

Claude finished @rockymadden's task in 1m 44s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdict counts: 0 Critical, 1 High, 0 Medium, 0 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body explicitly states: "Tests and evals were not run, so this PR makes no passing-run claim." Please consider editing the PR description to add a passing-run claim after verifying, e.g.: Ran skill-flow-coded-agent locally and it passed.

Mitigating context: The change adds only a weight: 0.0 / pass_threshold: 0.0 informational criterion that cannot affect scoring or gating. The author cites static analysis of test_same_ground_corpus.py, test_criterion_budgets.py, and coder_eval's weight-0 validator as evidence of correctness. Risk of breaking the existing task is minimal.

Per-task lint

tests/tasks/uipath-maestro-flow/single_node/coded_agent/coded_agent.yaml — verdict: OK

No issues on any rubric axis:

• Self-report anti-pattern: Not triggered. Criteria use run_command with Python check scripts, not agent-written report files.
• Prompt over-specification: Prompt states the goal (create a coded agent flow, validate it) without prescribing CLI flags or step-by-step procedure.
• Meaningful coverage: Two run_command criteria validate both structural correctness (flow validate, weight 3.0) and runtime correctness (debug returns approver count, weight 5.0). The new skill_triggered criterion at weight 0.0 is purely informational.
• Could pass for the wrong reason: Check scripts run flow validate and flow debug against the built artifact — not gameable without a working flow.
• Near-duplicate: Closest sibling is lowcode_agent.yaml — materially distinct operation (coded agent creation vs. published agent discovery/wiring, different node types).
• Validate-only flow tests: Not applicable — check_coded_agent_flow.py (line 55, timeout 720s) runs flow debug, not validate-only.
• CLI verb reachability: No command_executed criteria present; axis does not apply. (Script could not be run due to sandbox permissions — manual inspection confirms no command_pattern fields.)
• Redundant uip CLI in sandbox: No env_packages in sandbox config.
• Run-limit fields under agent: No agent: block; run_limits is top-level (line 17).

Within-PR duplicates

• No duplicate clusters detected (only 1 task YAML changed).

Conclusion

⚠ 1 task has issues, max severity High. The High is the missing passing-run claim only — the task content itself passes all rubric axes. Advisory only — not blocking merge.

---

[DevMomo]

Checked the YAML against coder_eval 0.12.8 (branch pin) and 0.12.14 (main pin) and ran the gates the body says were not run. The criterion itself is fine; one should-fix on the PR body.

What I ran / verified:

• pytest tests/tasks/uipath-maestro-flow/ on the branch: 1575 passed; on the merge-tree result with origin/main: 1744 passed. check-task-driver.py and check-task-host-paths.py --base-ref origin/main: OK. coder-eval plan at 0.12.8 and 0.12.14 (default, activation, smoke experiments): valid, only the pre-existing task_timeout 2400 > turn_timeout 1200 warning.
• test_gating_skill_telemetry_matches_the_temporary_allowlist (_shared/test_same_ground_corpus.py:274) only flags pass_threshold > 0, so no allowlist entry is needed.
• Budget: 180 + 720 + 30 = 930 < 2400. The 30 is the budgets test's own _DEFAULT_CRITERION_TIMEOUT (test_criterion_budgets.py:87); SkillTriggeredCriterion has no timeout field at all (timeout default 30 lives on RunCommandCriterion, criteria.py:396), so the sum is conservative.
• weight 0: check_weight_zero_is_not_gating (criteria.py:177) only rejects stop_early / suite_thresholds; is_gating = weight > 0; weighted_score excludes it (results.py:761). coder-eval evaluate exit code counts gating criteria only. This is the first weight-0 skill_triggered in the corpus; the other maestro-flow "Advisory:" ones use weight 1-3 + threshold 0 and so still move the weighted score on a no run. weight 0 is the right tool for a pure label, so no change asked.
• Regex replay (criteria/skill_triggered.py:39): .agents/skills/uipath/uipath-maestro-flow/SKILL.md -> uipath only; .agents/skills/uipath-maestro-flow/SKILL.md -> uipath-maestro-flow. supports_post_failure_evaluation is False on this type, so the "skipped after terminal failure" note holds (orchestrator.py:811, recorded with gating=False).
• Nothing reads criteria by position (coder_eval markdown.py slices per task; appending keeps old positions). No main commit touches this file since the merge-base; #3541 and #3725 do not touch it.
• The green "Run skill smoke tests" check filtered 0/1 tasks (no smoke tag), so CI has not exercised this criterion on a real run, as the body already says.

[DevMomo]

Approving. The diff is the same 6 lines we checked earlier (the new commit is only a merge from main). The weight-0 skill_triggered label is correct and passes the corpus gates. The earlier note about the PR body's out-of-date "why" (coder_eval#225 / #3982) still applies, but it doesn't block the merge.