From 1d3d818b2da30aaf845586430053f9e9a7c8f9fd Mon Sep 17 00:00:00 2001 From: roberts-cliff Date: Wed, 12 Aug 2026 16:15:01 -0600 Subject: [PATCH 1/4] fix(maestro-case): halt on a surviving placeholder connector-rule stub An unresolved `wait-for-connector` rule was allowed to ship. The skill emitted the Phase 2 placeholder stub, logged it as an ordinary Open Item, and reported the build complete. The resulting case is 100% non-startable. Connector-event subscriptions register at CASE START, before any stage is entered, so ONE surviving stub anywhere kills the whole case -- including when it sits on a non-required secondary stage nothing routes to. The failure surfaces as the case's own start event failing, which points nowhere near the offending rule, and `uip maestro case validate` reports `Valid` throughout. Measured on uip 1.198 (2026-08-12), same case, one field changed: rule uipath state start event elements validate connectorKey/operation="placeholder" Failed 1 Valid resolved connector (spliced shape) Completed 25 Valid Placeholder *tasks* (`data: {}`, Rule 8) are unaffected -- a case with placeholder tasks starts and runs. The fatal shape is specific to connector-bound condition rules. Adds Step 12 Check 14: every surviving stub halts the build, with no best-effort escape (unlike Checks 1/2/4/5/13, the artifact is not partially working -- it cannot start). Remediation is resolve / remove / abort; the abort path writes a literal BLOCKED marker to build-issues.md so the halt is a machine-readable side effect rather than prose. Also corrects the blast-radius claim in connector-trigger-impl.md ("the rule fails at debug/run" -> the case cannot start), revokes the "a remaining stub must map to a reported unresolved connector" permission in all four condition plugins, and adds Checks 13 and 14 to the Step 12 invocation list -- Check 13 was defined but never called. Test: tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/ Fixture SDD names a connector that exists on no tenant, so the unresolvable connector IS the fixture. The checker passes either legitimate resolution (no stub survives, plan stays coherent) or (stub survives AND the BLOCKED marker is present), and fails exactly the pre-fix outcome. Verified to discriminate against the real artifacts: the broken bug-bash case fails, the repaired variant passes. Co-Authored-By: Claude Opus 5 --- skills/uipath-maestro-case/SKILL.md | 4 +- .../references/connector-trigger-impl.md | 19 +- .../references/implementation.md | 20 +- .../references/placeholder-tasks.md | 4 +- .../case-exit-conditions/impl-json.md | 2 +- .../stage-entry-conditions/impl-json.md | 2 +- .../stage-exit-conditions/impl-json.md | 2 +- .../task-entry-conditions/impl-json.md | 2 +- .../check_connector_stub_halt.py | 217 ++++++++++++++ .../fixtures/sdd.md | 282 ++++++++++++++++++ .../unresolved_connector_rule_halt.yaml | 125 ++++++++ 11 files changed, 666 insertions(+), 13 deletions(-) create mode 100644 tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/check_connector_stub_halt.py create mode 100644 tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/fixtures/sdd.md create mode 100644 tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml diff --git a/skills/uipath-maestro-case/SKILL.md b/skills/uipath-maestro-case/SKILL.md index 56378282df..f16278278a 100644 --- a/skills/uipath-maestro-case/SKILL.md +++ b/skills/uipath-maestro-case/SKILL.md @@ -35,13 +35,13 @@ When `sdd.md` is absent, the case design belongs to **`uipath-planner` — the s 5. **Follow plugin per node type.** Open matching `planning.md` during planning + `impl-json.md` during execution. Never guess JSON shapes from memory. 6. **`tasks.md` declarative and lossless only.** No shell commands inside. Field names use plain identifiers (e.g., `type:`, `displayName:`, `lane:`), not CLI flag syntax. One T-entry per sdd.md declaration — every stage, task, trigger, condition, SLA rule, **variable, and argument** gets own T-number, even when value looks like default (`current-stage-entered`, `case-entered`, `exit-only`, `is-interrupting: false`, `runOnlyOnce: true`, `marks-stage-complete: true`). Never group, never silently omit. **An explicit stage/task entry or exit rule in a supplied or approved SDD is authoritative: planning and implementation preserve that exact rule and its selectors, even when a different rule would normally be inferred from task proximity or list order.** Preserve every stage/task/SLA `Design Rationale` and condition routing/activation rationale as `rationale:` on the matching T-entry; rationale is reviewer/audit context and never changes the executable JSON shape. Preserve every SDD Inputs row with its declared binding mode and value. A JSON object literal stays literal through both handoffs: record the exact JSON in `tasks.md`, then write either the native object or its JSON-encoded string to `input.value`; never add `=js:` or `=jsonString:` unless the SDD itself explicitly uses that prefix. Project every task/rule Outputs table row through the common grammar in [`plugins/variables/io-binding/planning.md`](references/plugins/variables/io-binding/planning.md#sdd-outputs-table-to-tasksmd-projection-mandatory), then preserve each resulting `outputs:` item **with its operator and both operands unchanged**. SDD Outputs rows require `->` or `=`; a bare `tasks.md` output is generated only from resolved-schema discovery and is never authored as an SDD row. SDD table placeholders such as a `—` Field are not operands and never appear in `tasks.md`. In particular, `greeting -> greeting` is NOT equivalent to schema-discovered bare `greeting`: the former extracts into the predeclared case variable and requires `originalVar`; the latter auto-mints a task-local output. Never simplify an equal-name `->` row. **When an sdd.md row's format is unrecognized, ambiguous, or cannot be categorized — invoke AskUserQuestion before skipping. Silent omission is forbidden.** Always regenerate from scratch (greenfield/planning only — brownfield targeted edits mutate in place and preserve IDs; see [references/brownfield.md](references/brownfield.md)). **Every §4.6 task T-entry carries its own `activation-mode:` and `entry-rule:` lines — a separate §4.7 `rule-type:` entry does not satisfy this.** **Every task T-entry heading quotes the task's display name** in the exact form `## T: Add task "" to ""` (e.g. `## T08: Add wait-for-timer task "First Step" to "Process"`) — an unquoted or reworded heading (e.g. `## T08: Task First Step`) breaks plan addressability and fails plan validators even when `caseplan.json` itself is correct. See [`references/planning.md` §4.0](references/planning.md) and the [Plan-shape gate](references/planning.md#step-5--finalize-tasksmd-auto-proceed-to-phase-2). 7. **`tasks.md` gate — auto-approved by default, opt-in stop.** Phase 1 auto-proceeds into Phase 2 Prototyping with no AskUserQuestion sign-off; treat the plan as approved. **Stop after `tasks.md` only when the request explicitly asked for a plan-only / review-first run** (e.g. "just the plan", "Phase 1 only", "stop after tasks.md for review", "don't build the case yet") — then report the plan and do NOT proceed to Phase 2. Re-read `tasks.md` before executing. -8. **Unresolved resource → placeholder, never fabricate IDs.** Keep `` markers in `tasks.md`. Placeholder **task**: node with `type` + `displayName` + structural fields, `data: {}`; conditions still reference the TaskId. Placeholder **event trigger**: node with render fields + `data.inputs: { serviceType: "Intsvc.EventTrigger" }` only (no other `data.inputs` keys); `entry-points.json` entry appended. No trigger-edge is created (Rule 20). See [references/placeholder-tasks.md](references/placeholder-tasks.md) and [references/plugins/triggers/event/impl-json.md § Placeholder fallback](references/plugins/triggers/event/impl-json.md). +8. **Unresolved resource → placeholder, never fabricate IDs.** Keep `` markers in `tasks.md`. Placeholder **task**: node with `type` + `displayName` + structural fields, `data: {}`; conditions still reference the TaskId. Placeholder **event trigger**: node with render fields + `data.inputs: { serviceType: "Intsvc.EventTrigger" }` only (no other `data.inputs` keys); `entry-points.json` entry appended. No trigger-edge is created (Rule 20). **Exception — a connector-bound condition-rule stub may NOT survive the build.** Placeholder tasks and triggers are shippable build state; a `wait-for-connector` rule still carrying `connectorKey`/`operation` = `"placeholder"` after Phase 3 is **case-fatal** — subscriptions register at case start, so one stub anywhere makes the ENTIRE case non-startable (start event fails, 1 element executed) even when it sits on a secondary stage nothing routes to, and `validate` reports `Valid` throughout. Step 12 **Check 14** halts the build on any survivor with no best-effort escape. See [references/placeholder-tasks.md](references/placeholder-tasks.md), [references/connector-trigger-impl.md § Placeholder fallback](references/connector-trigger-impl.md#placeholder-fallback), and [references/plugins/triggers/event/impl-json.md § Placeholder fallback](references/plugins/triggers/event/impl-json.md). 9. **Persist every registry resolution to `registry-resolved.json`** — one object per task with exact keys `stage`, `task`, `taskType`, `cacheFile`, `searchQuery`, `matches`, `selected`, and `rationale` (plus resolved I/O/review metadata, and `gateDecision` ONLY when the user answered the design-time gate for the resource — defaulted deferrals carry none, keeping them eligible for the Rule 17 gate). `stage` + `task` associate the audit entry to one SDD declaration; `cacheFile` is the basename actually searched; `matches` is the full exact-name match set from the cache refreshed in Rule 3, not a summary. Use the authoritative SDD fields as the search and selection contract; record `selected` from that match set, or `null` after a genuine empty lookup. **Same-session planner design:** the lane's resolution ledger (in context from the design conversation) already carries this exact shape — persist it verbatim as the file's initial content, then verify/extend per Rule 3. 10. **Cross-task refs:** plan as `"Stage Name"."Task Name".output_name`. Resolve both whole-value `<-` and in-expression `$xref` through the common output-reference-ID algorithm in [`plugins/variables/io-binding/impl-json.md`](references/plugins/variables/io-binding/impl-json.md#output-reference-id-authoritative): use the source output's `.id`; only a custom `=` output, which intentionally has no `.id`, resolves through its verified root companion's `.id`. Never use a reassigned output's `.var` as the source reference ID — it points at the target Case variable and can differ from the collision-safe source `.id`. Discover output names via `uip maestro case spec` (connector tasks) or `uip maestro case tasks describe` (non-connector tasks) — never fabricate. **Inside** a larger `=js:` expression (composite payload, condition, SLA), use the in-expression marker `vars.$xref('Stage','Task','output')` instead — resolved at Step 11.5. See [references/bindings-and-expressions.md](references/bindings-and-expressions.md) and [`plugins/variables/io-binding/impl-json.md`](references/plugins/variables/io-binding/impl-json.md). 11. **Build-review preference decides the Phase 2 → Phase 3 boundary — captured ONCE, up front, never asked mid-build.** Capture it at journey start: greenfield-with-design-handoff folds it into the lane's Case Review Build options (`Build it — straight through` / `Build it — pause at the build preview` — Rule 1); greenfield-with-provided-SDD asks it once right after the roadmap; non-interactive runs and resumed runs with no recorded preference default to **straight-through** (no mid-build publish — Phase 5 stays the only publish point and stays gated). At the boundary, try `validate --skeleton-v2` first; fall back once to legacy `--skeleton` **only** when the parser response names `--skeleton-v2` as unknown or unsupported (typically `ErrorCode: invalid_argument` and exit 3). Exit 3 alone is not enough. A genuine validation failure means v2 ran — report it and do not hide it with a fallback. Name the selected profile in the counts summary; the gate is advisory and never halts on validation findings. Then: **straight-through** → continue into Phase 3 with no prompt, the summary line doubling as the milestone narration; **pause-at-preview** → follow the publish-for-review contract in [`references/phased-execution.md`](references/phased-execution.md) (AskUserQuestion `Publish for review` / `Skip publish and continue` / `Abort`; on publish, print `DesignerUrl` as plain text BEFORE the follow-up prompt — never only inside the question body). Hard stops that are NEVER bypassed regardless of preference: Phase 4 retry exhaustion (`Retry with fix` / `Pause for manual edit` / `Abort`), Phase 5 entry (`Publish to Studio Web` / `Skip to Debug`), Phase 6 entry (`Run debug session` / `Continue to publish`), and Phase 7 entry (`Publish to Orchestrator` / `Done`). Full contract in [`references/phased-execution.md`](references/phased-execution.md). 12. **Never run `uip maestro case debug` or the Phase 7 Orchestrator publish automatically.** `debug` executes the case for real — emails, messages, API calls. The Phase 7 chain (`uip solution pack` → `uip solution publish`) ships the case to the tenant solution feed. Both require explicit user consent via their own AskUserQuestion gate. 13. **All skill artifacts: Read + Write/Edit only.** Applies to `caseplan.json`, `sdd.md`, `sdd.draft.md`, `tasks.md`, `tasks/registry-resolved.json`, `tasks/trigger-spec-cache.json`, `tasks/spec-cache..json`, `bindings_v2.json`, `id-map.json`, `entry-points.json`, `build-issues.md`. No `python`, `node`, `jq`, `sed`, `awk`, or scripts that open/parse/modify/save these files. **Shell filesystem commands may not create, replace, rename, or relocate an artifact** — `cp`, `mv`, `install`, and `rsync` are forbidden for these files, including copying or renaming `sdd.draft.md` to `sdd.md`. **Specifically forbidden** (common slip): `node -e "...fs.writeFileSync..."`, `node -e "...fs.readFileSync..."`, `node -e "..." > `, `jq '...' > `, `python -c "...open(...,'w')..."`, `sed -i`, `awk -i inplace`, or any shell redirection (`>`, `>>`, `| tee`) onto a skill artifact regardless of interpreter. **Writing a helper script under `/tmp` or anywhere else to assemble a skill artifact is also forbidden** — the build-assembler pattern (`/tmp/build-caseplan.js`, `/tmp/gen-tasks.py`, etc.) is the same Rule 13 violation as inline `node -e`, regardless of "mechanical copy" or "avoid Read+Write churn" framing. If `caseplan.json` exceeds ~30KB and a single Write feels too large, split into the Phase-2-preview-then-Phase-3-detail cadence (per [case-editing-operations.md § Per-section batch write contract](references/case-editing-operations.md#per-section-batch-write-contract--canonical)) — never via helper script. **The `node -e ... fs.*` ban is not scoped to the artifact list — it applies to ALL file reads in this skill, including resource cache reads from `~/.uip/case-resources/`. Use `cat ... | python3 -c "..."` or the `Read` tool for cache lookups.** Bash subprocesses OK ONLY for UUID v4 generation (`node -e "console.log(crypto.randomUUID())"` for `operate.json.projectId` and `entry-points.json` `uniqueId` — subprocess MUST NOT `require('fs')` or use redirection), CLI metadata fetches, validate, debug, and solution scaffold/upload. **Prefixed IDs (`Stage_`, `t`, `Rule_`, etc.) are picked inline by the agent — no subprocess.** See [references/case-editing-operations.md § Tool usage](references/case-editing-operations.md#tool-usage--mandatory). -14. **Resolved resources must be runnable, and sidecar parity is an unconditional Phase 3 exit check.** Before Phase 4, run Step 12 Checks 7, 9, 11, and 12 even when publish, debug, and `uip solution resources refresh` are skipped. A task with a non-null `selected` entry in `tasks/registry-resolved.json` MUST NOT be emitted as a placeholder: it must remain present with `data.name` and `data.folderPath` bound to complete root bindings, its resource must project into `bindings_v2.json.resources[]`, and its binding pair's `resourceKey` must be self-consistent with its own `name`/`folderPath` defaults (Check 11) — never a copied tenant identity/UUID. Checks 9 and 11 exempt connector nodes; **Check 12** covers them — a resolved connector task, event trigger, or `wait-for-connector` rule must carry the spliced `caseShape.context` plus its Connection/Folder root bindings, never the `typeId`+`connectionId`-only degraded shape. `uip maestro case validate` success does not substitute for these checks. On a mismatch, repair the named task/binding or regenerate the sidecar once as applicable, then re-check; halt before Phase 4 if any of these checks still fails. Repeat Check 7 before every `resources refresh`. Always run `resources refresh` before `uip solution upload` or `uip maestro case debug` so Studio Web can resolve dependencies. Every `uip solution upload` call MUST carry `--output-filter "{Status: Status, SolutionId: SolutionId, DesignerUrl: DesignerUrl}"` — the unfiltered response gets truncated and `DesignerUrl` is lost. +14. **Resolved resources must be runnable, and sidecar parity is an unconditional Phase 3 exit check.** Before Phase 4, run Step 12 Checks 7, 9, 11, 12, and 14 even when publish, debug, and `uip solution resources refresh` are skipped. A task with a non-null `selected` entry in `tasks/registry-resolved.json` MUST NOT be emitted as a placeholder: it must remain present with `data.name` and `data.folderPath` bound to complete root bindings, its resource must project into `bindings_v2.json.resources[]`, and its binding pair's `resourceKey` must be self-consistent with its own `name`/`folderPath` defaults (Check 11) — never a copied tenant identity/UUID. Checks 9 and 11 exempt connector nodes; **Check 12** covers them — a resolved connector task, event trigger, or `wait-for-connector` rule must carry the spliced `caseShape.context` plus its Connection/Folder root bindings, never the `typeId`+`connectionId`-only degraded shape. `uip maestro case validate` success does not substitute for these checks. On a mismatch, repair the named task/binding or regenerate the sidecar once as applicable, then re-check; halt before Phase 4 if any of these checks still fails. Repeat Check 7 before every `resources refresh`. Always run `resources refresh` before `uip solution upload` or `uip maestro case debug` so Studio Web can resolve dependencies. Every `uip solution upload` call MUST carry `--output-filter "{Status: Status, SolutionId: SolutionId, DesignerUrl: DesignerUrl}"` — the unfiltered response gets truncated and `DesignerUrl` is lost. 15. **Design handoff contract — invoke the `uipath-planner` Case Design Lane in THIS conversation; never a subagent.** When `sdd.md` is absent at the resolved path (or the user asks to finalize a case `sdd.draft.md`), load the `uipath-planner` skill and enter its Case Design Lane in **Build handoff** mode: the lane runs here, in the same context window, with the same user — the in-memory model, the resolution ledger, and the session's registry pull all cross the skill boundary for free. Hand off immediately on detection — before reading this skill's references, hunting the filesystem for templates, or running tenant commands (the lane owns tenant grounding). The lane owns everything through the Case Review's Build answer and the `sdd.md` write; this skill resumes at the first build action. User-facing language never mentions the handoff — the user sees one continuous flow. If `uipath-planner` is unavailable in this environment, say so in one line, ask the user to provide an `sdd.md` (or approve building from a design they paste), and stop — never improvise an interview, never fabricate an SDD. Cross-product planning beyond the case (multi-project solutions) stays a plain-text suggestion of `uipath-planner` — no handoff for that. **Receipt spot-check (one Grep — run it BEFORE Phase 1 reads any `sdd.md` this conversation did not watch being written, e.g. user-provided files):** the file must contain the four blueprint headings (`## Section 1: Case Definition` through `## Section 4: Integrations`) and at least one `##### Task` detail block. A freeform/summary SDD → do not build on it; route it back through the lane's template conformance gate. 16. **Caseplan task `type` enum is closed — 9 values, schema-kebab.** Any task node written into `caseplan.json` MUST have `type` exactly one of: `process` | `agent` | `rpa` | `action` | `api-workflow` | `case-management` | `execute-connector-activity` | `wait-for-connector` | `wait-for-timer`. **Never** write the plugin folder name (`connector-activity`, `connector-trigger`) or the CLI `--type` flag value into the JSON node — those name the planning artifacts, not the schema. Never write `external-agent`, `external-workflow`, `document-extraction`, `flow-process`, `wait-for-event`, or any hallucinated value — there is no plugin to back them. `external-agent`, `external-workflow`, `document-extraction`, and `flow-process` are **not supported yet**. See [references/case-schema.md § Task type](references/case-schema.md) and the Plugin Index naming-asymmetry table below. 17. **Empty registry lookup → AskUserQuestion BEFORE any placeholder fallback — unless the USER already ruled at the design-time gate.** When the same-session resolution ledger carries a `gateDecision` for the resource (`resolve-at-build` → placeholder; `create-during-build` → the inline-create flow below; `pick:` → bind that entry), execute it without re-asking — the gate ran once, at the planner's Case Review, and `gateDecision` exists only for items the user actually answered there. **An entry with NO `gateDecision` is a defaulted deferral** (no session, failed/pending pull, or non-interactive design) — it gets the full gate below, exactly like any fresh empty lookup; never treat a defaulted `resolve at build` as a user decision. Otherwise, when a planning-phase lookup returns 0 matches, present AskUserQuestion per lookup-batch (one prompt, not per-task) BEFORE any placeholder T-entry or per-plugin Unresolved Fallback, with options: (a) `Force pull and re-resolve` — loops back for still-empty; (b) `Use placeholders for all`; (c) `Create missing resources inline` — shown ONLY when ≥1 still-empty is creatable (an `agent` or an `api-workflow`) AND the CLI supports `registry --local`. **Create covers agents and API workflows only, gate-selected only** (never from SDD content alone; agent → `uipath-agents`, api-workflow → `uipath-api-workflow`); unselected + non-creatable empties (regular RPA process, action, case-management, connectors, agentic processes) → placeholder; the option is suppressed when `--local` is absent. Do NOT pre-judge via resource-name heuristics — the user's call. The gate and Select **group empties by `(name, type)`** (one row per resource, usages listed — non-creatable ones show only at the gate, annotated `placeholder only`); create-selected resources merge or split at [registry-discovery § 1c](references/registry-discovery.md#1c--dedup-the-selected-builds-one-resource-per-name-and-type) by I/O (identical-I/O usages → one build; differing → later renamed, anchor keeps the name; the SDD cell is updated only with user permission — never non-interactively). Placeholder fallback is valid only after `Use placeholders for all`. Build/register/verify mechanics live in [references/registry-discovery.md § Create-on-Missing](references/registry-discovery.md#create-on-missing-build-and-rediscovery) (gate detail: [§ MUST Confirm](references/registry-discovery.md#must-confirm-before-placeholder-fallback)). diff --git a/skills/uipath-maestro-case/references/connector-trigger-impl.md b/skills/uipath-maestro-case/references/connector-trigger-impl.md index 23f9b480ca..db2f8b2180 100644 --- a/skills/uipath-maestro-case/references/connector-trigger-impl.md +++ b/skills/uipath-maestro-case/references/connector-trigger-impl.md @@ -135,9 +135,9 @@ A `wait-for-connector` rule inside a condition (`…conditions[].rules[i][j]`) b ### Condition-rule phase contract - **Phase 2 Step 10:** write every `wait-for-connector` rule with the canonical stub from [§ Placeholder fallback](#placeholder-fallback), even when the connector resolved in planning. The enclosing condition, rule ID, expression, scope, and placement are final at this point. Do not run `case spec`, add connector bindings, or dispatch outputs. -- **Phase 3 Step 10.5:** for a resolved connector, run the procedure below and replace only `rule.uipath`. Preserve all enclosing Phase 2 state. For an unresolved connector or failed `case spec`, keep the stub, log it, and report it as not runnable. +- **Phase 3 Step 10.5:** for a resolved connector, run the procedure below and replace only `rule.uipath`. Preserve all enclosing Phase 2 state. For an unresolved connector or failed `case spec`, keep the stub and log it — then **halt at Check 14** (see [§ Placeholder fallback](#placeholder-fallback)); a surviving stub is a build failure, not a reportable open item. -The same stub therefore has two lifetimes: temporary for a resolved connector awaiting Phase 3, permanent for an unresolved connector. Only the permanent case is an unresolved-resource issue. +The same stub therefore has two lifetimes: temporary for a resolved connector awaiting Phase 3, permanent for an unresolved connector. The permanent case is not merely an unresolved-resource issue — it makes the whole case non-startable, so it is the one unresolved-resource class the skill must NOT ship. ### Procedure (Phase 3) @@ -197,7 +197,20 @@ Emit a **stub `uipath`**, never a bare rule. The stub is the minimum shape accep } ``` -This stub is a **deliberate mock**. While temporary, it is simply Phase 2 build state. If it remains after Phase 3, Studio Web flags it and the rule **fails at debug/run**. A remaining stub has no real outputs, Connection/Folder bindings, IS-cache entry, or rule-specific `bindings_v2` resource. Stamp unresolved `tasks.md` entries with Rule 8 markers, log them, and list them in the completion report as **"replace the `placeholder` connector values before debug / publish-to-run."** Upgrade later by re-running the [§ Procedure](#procedure-phase-3). +This stub is a **deliberate mock**, and it is safe ONLY while temporary — as Phase 2 build state awaiting the Phase 3 upgrade. + +> **A stub that survives Phase 3 is case-fatal, not rule-local.** Connector-event subscriptions are registered at **case start**, before any stage is entered, so ONE unresolved stub anywhere in the case makes the ENTIRE case non-startable — including stages that never reference it. The failure surfaces as the case's own start event failing (`trigger_1` → `Failed`, `finalStatus: Faulted`, **1 element executed**), which looks nothing like the rule that caused it. `uip maestro case validate` reports `Valid` throughout. Never emit a surviving stub and call the build complete — halt per [implementation.md § Step 12 Check 14](implementation.md#step-12--end-of-phase-3-validator-pass). +> +> Measured on `uip 1.198`, same case, one field changed (2026-08-12): +> +> | Rule `uipath` state | Start event | Elements executed | `validate` | +> |---|---|---|---| +> | `connectorKey`/`operation` = `"placeholder"` | **Failed** | **1** | `Valid` | +> | resolved connector (spliced `caseShape`) | Completed | 25 | `Valid` | +> +> Placeholder **tasks** (`data: {}`, Rule 8) are NOT affected — a case with placeholder tasks starts and runs normally. The fatal shape is specific to connector-bound **condition rules**. + +A remaining stub has no real outputs, Connection/Folder bindings, IS-cache entry, or rule-specific `bindings_v2` resource. Stamp unresolved `tasks.md` entries with Rule 8 markers and log them, but do NOT downgrade the finding to a completion-report note — Check 14 halts the build. Upgrade by re-running the [§ Procedure](#procedure-phase-3). --- diff --git a/skills/uipath-maestro-case/references/implementation.md b/skills/uipath-maestro-case/references/implementation.md index eb1d696a7e..fca6734289 100644 --- a/skills/uipath-maestro-case/references/implementation.md +++ b/skills/uipath-maestro-case/references/implementation.md @@ -265,7 +265,7 @@ Read `caseplan.json` and scan all four condition scopes for `wait-for-connector` For each matched rule whose connector resolved in planning, run the connector-trigger `case spec --type trigger --input-details` procedure, mint its output IDs/element IDs, and gather its root Connection/Folder bindings. Then Edit **only that rule's `uipath` block**. Preserve the enclosing condition array plus the rule's `id`, `rule`, `conditionExpression`, scope, and placement. Apply declared rule-output bindings after the real outputs exist. -If the connector is `` or `case spec` fails, leave the stub unchanged, log it, and list it in the completion report. After all successful upgrades, populate the IS cache and regenerate `bindings_v2.json` once. Re-scan: every resolved rule must be free of `"placeholder"`; any remaining stub must map to a reported unresolved connector. Full procedure and scope-specific `elementId` rules: [`connector-trigger-impl.md § Target: connector-bound condition rule`](connector-trigger-impl.md#target-connector-bound-condition-rule). +If the connector is `` or `case spec` fails, leave the stub unchanged and log it — then let **Check 14** halt the build. Do NOT list it in the completion report and proceed: a surviving stub makes the whole case non-startable ([connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback)). After all successful upgrades, populate the IS cache and regenerate `bindings_v2.json` once. Re-scan: every rule must be free of `"placeholder"`. Full procedure and scope-specific `elementId` rules: [`connector-trigger-impl.md § Target: connector-bound condition rule`](connector-trigger-impl.md#target-connector-bound-condition-rule). ## Step 11.5 — Resolve in-expression `vars.$xref` markers (whole-file pass) @@ -275,7 +275,7 @@ Runs after bindings (9.8) and connector-rule upgrades (10.5), when every task / > **Algorithm reference:** the per-check pseudocode + AskUserQuestion prompt templates + skill-response-per-pick details all live in [`plugins/variables/io-binding/impl-json.md § Binding Procedure`](plugins/variables/io-binding/impl-json.md#binding-procedure). This step is the orchestration hook; that doc is the algorithm. When in doubt, follow the impl-json doc. -After value bindings (Step 9.8), connector-rule upgrades (Step 10.5), and marker resolution (Step 11.5), invoke the end-of-Phase-3 validator — Checks 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Phase 2 conditions and SLA remain in place throughout. +After value bindings (Step 9.8), connector-rule upgrades (Step 10.5), and marker resolution (Step 11.5), invoke the end-of-Phase-3 validator — Checks 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Phase 2 conditions and SLA remain in place throughout. - **Check 1** — Resolve every `=vars.X` reference against `variables.{inputs, inputOutputs}[].id`. Scan all task input `value` fields, entry/exit condition expressions (stage and task), case-exit and trigger rule expressions, SLA expressions, and `=js:` expressions anywhere they appear. On unresolved → **AskUserQuestion** offering: (a) name the intended variable, (b) remove the reference, (c) continue with best-effort emit (entry logged under Open Items, runtime returns undefined). - **Check 2 — Out-arg producer presence** — For every formal Out-arg in `variables.outputs[]`, verify the producer/Default situation per [`io-binding/impl-json.md` § Check 2](plugins/variables/io-binding/impl-json.md): @@ -302,7 +302,21 @@ After value bindings (Step 9.8), connector-rule upgrades (Step 10.5), and marker - **Check 13 — Rule task-selector completeness** — Enumerate every rule across all 4 condition scopes (stage-entry / stage-exit / task-entry, plus case-exit under `metadata.caseExitRules`) whose rule type requires a task selector (`selected-tasks-completed`). Each MUST carry a non-empty `selectedTasksIds` array in which every id resolves to a task in the owning stage. **Non-interactive repair:** resolve the missing ids from the rule's T-entry `selected-tasks-ids` names via `tasks/id-map.json` using EXACT `tasks.md` display names (paraphrased or shortened names are the common miss — match the task's exact name, per the conditions plugins' selector contract); rewrite the rule, re-scan. Unresolvable after one pass → **AskUserQuestion** (name the intended task / repair the plan / continue with best-effort emit, logged under Open Items). Halt before Phase 4 while any selector is empty with no user decision. `uip maestro case validate` reports these as `... has no task(s) selected` — that finding surfacing in Phase 4 means this check was skipped. -**Build-with-best policy:** for any user pick of "continue with best-effort emit" on a Check 1, Check 2, Check 4, Check 5, or Check 13 AskUserQuestion, append a `## Open Items for User` entry to `tasks/build-issues.md` and proceed to Phase 4. AskUserQuestion is the surface; build-with-best is the escape. The skill conservatively emits what it has; Phase 4 validate stays green (structural validity is intact); runtime concerns are listed for pre-publish review. +- **Check 14 — No surviving connector-rule placeholder stub (CASE-FATAL, halts)** — Scan every rule across all 4 condition scopes (stage-entry / stage-exit / task-entry, plus case-exit under `metadata.caseExitRules`) for a `wait-for-connector` rule whose `uipath.context` still carries `connectorKey: "placeholder"` / `operation: "placeholder"`. **Every survivor is a build failure**, independent of its registry entry — this check does NOT exempt `selected: null` the way Check 12 does, because an unresolved connector rule is exactly the fatal case. + + **Why it halts rather than degrading.** Connector-event subscriptions register at **case start**, before any stage is entered. One surviving stub anywhere makes the ENTIRE case non-startable — including when it sits on a non-required secondary stage that nothing routes to. The observable failure is the case's own start event failing (`finalStatus: Faulted`, **1 element executed**), which points nowhere near the offending rule. `uip maestro case validate` reports `Valid` at every step, so this check is the ONLY build-time gate. Measured evidence + the one-field control/variant table: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). + + **Remediation — AskUserQuestion, no best-effort option.** Name the stage, condition, and rule, then offer exactly three picks: (a) **resolve the connector now** — supply/choose the connector + connection, re-run Step 10.5 for that rule, re-scan; (b) **remove the rule** — delete it and, when it was the condition's only rule, the condition too (if that leaves a stage with no entry condition, remove the stage and every reference to it, then re-run Checks 8 and 13); (c) **abort the build** — leave `caseplan.json` on disk and stop. + + **On abort (or any exit while a stub survives), `tasks/build-issues.md` MUST carry a literal line of this exact form**, above `## Open Items for User`: + + ``` + BLOCKED: case is not startable — unresolved wait-for-connector rule(s) still carry the placeholder stub. + ``` + + Write `rule(s)` literally, regardless of N — the marker is matched mechanically, so do not pluralize it. List each offending `(scope, stage, condition, rule id, connector)` beneath it. This line is the machine-readable halt marker: its presence means "an artifact exists but must not be run or published." The completion report MUST reproduce it verbatim and MUST NOT describe the build as complete, successful, or ready. "Continue with best-effort emit" is NOT offered: unlike Checks 1/2/4/5/13, the artifact is not partially working — it cannot start at all, so emitting it produces a case that is 100% dead while reporting success. **Never enter Phase 4, publish, debug, or report completion while any stub survives.** + +**Build-with-best policy:** for any user pick of "continue with best-effort emit" on a Check 1, Check 2, Check 4, Check 5, or Check 13 AskUserQuestion, append a `## Open Items for User` entry to `tasks/build-issues.md` and proceed to Phase 4. Check 14 is deliberately excluded — it has no best-effort escape. AskUserQuestion is the surface; build-with-best is the escape. The skill conservatively emits what it has; Phase 4 validate stays green (structural validity is intact); runtime concerns are listed for pre-publish review. **Reporting:** at end of Phase 4, count entries in the `## Open Items for User` section of `tasks/build-issues.md` (read the file after writing). If count > 0, the completion report MUST include a literal line of the form: diff --git a/skills/uipath-maestro-case/references/placeholder-tasks.md b/skills/uipath-maestro-case/references/placeholder-tasks.md index c20345c8bf..6543c6a6f5 100644 --- a/skills/uipath-maestro-case/references/placeholder-tasks.md +++ b/skills/uipath-maestro-case/references/placeholder-tasks.md @@ -77,7 +77,9 @@ Case-level event triggers (`type: "uipath.case.trigger"` with `data.inputs.servi ### Connector condition rules -When a `wait-for-connector` rule's connector hasn't resolved at write-time, emit the rule with a **stub `uipath`** (`serviceType` + 2 `"placeholder"` context fields: `connectorKey` + `operation`) — a deliberate mock that validates clean but fails at Studio Web / debug / run until replaced. Full recipe + skip behavior + upgrade path: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). +When a `wait-for-connector` rule's connector hasn't resolved at write-time, emit the rule with a **stub `uipath`** (`serviceType` + 2 `"placeholder"` context fields: `connectorKey` + `operation`). Full recipe + skip behavior + upgrade path: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). + +> **This is the one placeholder the skill may NOT ship.** Unlike a placeholder task, a surviving connector-rule stub is **case-fatal**: subscriptions register at case start, so one stub anywhere makes the entire case non-startable (start event fails, 1 element executed) while `validate` still reports `Valid`. It is legal as Phase 2 build state only. If it survives Phase 3, [Step 12 Check 14](implementation.md#step-12--end-of-phase-3-validator-pass) halts the build — it is never downgraded to an Open Item. Evidence table: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). ## `tasks.md` Planning-Entry Shape diff --git a/skills/uipath-maestro-case/references/plugins/conditions/case-exit-conditions/impl-json.md b/skills/uipath-maestro-case/references/plugins/conditions/case-exit-conditions/impl-json.md index 2ece30109e..ca66b8e2c9 100644 --- a/skills/uipath-maestro-case/references/plugins/conditions/case-exit-conditions/impl-json.md +++ b/skills/uipath-maestro-case/references/plugins/conditions/case-exit-conditions/impl-json.md @@ -80,6 +80,6 @@ In Phase 2, always write the canonical stub from [connector-trigger-impl.md § C Confirm `metadata.caseExitRules[]` contains the new object with `id`, non-empty `displayName` (SDD value or `Complete Rule {N}` / `Exit Rule {N}` default keyed to `marksCaseComplete`), `marksCaseComplete` matching the T-entry, and `rules` carrying the expected `rule` value plus any required side field. Verify no `root` key exists at the top level. -For `wait-for-connector`, Phase 2 verification expects the exact two-entry placeholder context plus empty inputs/outputs/bindings. After Phase 3, a resolved rule must have no `"placeholder"` values, inputs/outputs must use `root-`, and ConnectionId + FolderKey root bindings must exist; a remaining stub must map to a reported unresolved connector. +For `wait-for-connector`, Phase 2 verification expects the exact two-entry placeholder context plus empty inputs/outputs/bindings. After Phase 3, EVERY such rule must have no `"placeholder"` values, inputs/outputs must use `root-`, and ConnectionId + FolderKey root bindings must exist. **A stub surviving Phase 3 is case-fatal, not a reportable open item** — it makes the whole case non-startable and halts the build at [implementation.md § Step 12 Check 14](../../../implementation.md#step-12--end-of-phase-3-validator-pass). diff --git a/skills/uipath-maestro-case/references/plugins/conditions/stage-entry-conditions/impl-json.md b/skills/uipath-maestro-case/references/plugins/conditions/stage-entry-conditions/impl-json.md index c378004014..b0148420c5 100644 --- a/skills/uipath-maestro-case/references/plugins/conditions/stage-entry-conditions/impl-json.md +++ b/skills/uipath-maestro-case/references/plugins/conditions/stage-entry-conditions/impl-json.md @@ -125,6 +125,6 @@ In Phase 2, always write the canonical stub from [connector-trigger-impl.md § C ## Post-Write Verification -Confirm target stage's `data.entryConditions[]` contains the new object with `id`, non-empty `displayName` (SDD value or `Entry Rule {N}` default), `isInterrupting` matching the T-entry, and `rules` carrying the expected `rule` value plus any required side field. For `sla-status-change`, verify `slaId` resolves to an object on the declared SLA target and `isInterrupting` matches the declared response. Check `escalationId` **against the T-entry's status, not for its presence**: an at-risk T-entry must carry one that resolves on that same SLA; a breach T-entry must carry **none**. A missing `escalationId` on a breach rule is correct and complete — do **not** "repair" it by adding one, which silently converts the rule to at-risk ([sla-response-shapes.md § defects `validate` cannot see](../../../sla-response-shapes.md)). For `wait-for-connector`, Phase 2 expects the exact stub; after Phase 3, a resolved rule must have no `"placeholder"` values, use `-` on inputs/outputs, and carry root bindings. A remaining stub must map to a reported unresolved connector. +Confirm target stage's `data.entryConditions[]` contains the new object with `id`, non-empty `displayName` (SDD value or `Entry Rule {N}` default), `isInterrupting` matching the T-entry, and `rules` carrying the expected `rule` value plus any required side field. For `sla-status-change`, verify `slaId` resolves to an object on the declared SLA target and `isInterrupting` matches the declared response. Check `escalationId` **against the T-entry's status, not for its presence**: an at-risk T-entry must carry one that resolves on that same SLA; a breach T-entry must carry **none**. A missing `escalationId` on a breach rule is correct and complete — do **not** "repair" it by adding one, which silently converts the rule to at-risk ([sla-response-shapes.md § defects `validate` cannot see](../../../sla-response-shapes.md)). For `wait-for-connector`, Phase 2 expects the exact stub; after Phase 3, EVERY such rule must have no `"placeholder"` values, use `-` on inputs/outputs, and carry root bindings. **A stub surviving Phase 3 is case-fatal, not a reportable open item** — it makes the whole case non-startable and halts the build at [implementation.md § Step 12 Check 14](../../../implementation.md#step-12--end-of-phase-3-validator-pass). diff --git a/skills/uipath-maestro-case/references/plugins/conditions/stage-exit-conditions/impl-json.md b/skills/uipath-maestro-case/references/plugins/conditions/stage-exit-conditions/impl-json.md index b99999be90..da60f2cf36 100644 --- a/skills/uipath-maestro-case/references/plugins/conditions/stage-exit-conditions/impl-json.md +++ b/skills/uipath-maestro-case/references/plugins/conditions/stage-exit-conditions/impl-json.md @@ -130,6 +130,6 @@ The exception lane's entry is `selected-stage-exited("") + IF =js:(vars. ## Post-Write Verification -Confirm target stage's `data.exitConditions[]` contains the new object with `id`, non-empty `displayName` (SDD value or `Complete Rule {N}` / `Exit Rule {N}` default keyed to `marksStageComplete`), `type`, `exitToStageId` (if set), `marksStageComplete` matching the T-entry, and `rules` carrying the expected `rule` value plus any required side field. For `wait-for-connector`, Phase 2 expects the exact stub; after Phase 3, a resolved rule must have no `"placeholder"` values, use `-` on inputs/outputs, and carry root bindings. A remaining stub must map to a reported unresolved connector. +Confirm target stage's `data.exitConditions[]` contains the new object with `id`, non-empty `displayName` (SDD value or `Complete Rule {N}` / `Exit Rule {N}` default keyed to `marksStageComplete`), `type`, `exitToStageId` (if set), `marksStageComplete` matching the T-entry, and `rules` carrying the expected `rule` value plus any required side field. For `wait-for-connector`, Phase 2 expects the exact stub; after Phase 3, EVERY such rule must have no `"placeholder"` values, use `-` on inputs/outputs, and carry root bindings. **A stub surviving Phase 3 is case-fatal, not a reportable open item** — it makes the whole case non-startable and halts the build at [implementation.md § Step 12 Check 14](../../../implementation.md#step-12--end-of-phase-3-validator-pass). diff --git a/skills/uipath-maestro-case/references/plugins/conditions/task-entry-conditions/impl-json.md b/skills/uipath-maestro-case/references/plugins/conditions/task-entry-conditions/impl-json.md index fdbbecd4ab..cfb41f209f 100644 --- a/skills/uipath-maestro-case/references/plugins/conditions/task-entry-conditions/impl-json.md +++ b/skills/uipath-maestro-case/references/plugins/conditions/task-entry-conditions/impl-json.md @@ -149,6 +149,6 @@ When a *stage* should take the case instead, the rule goes on the stage's `entry ## Post-Write Verification -Confirm target task's `entryConditions[]` length equals the number of task-entry T-tasks tasks.md wrote for this task. Each entry carries `id` (prefix `c`), non-empty `displayName` (SDD value or `Entry Rule {N}` default), and `rules` with the expected `rule` value plus any required side field. For `wait-for-connector`, Phase 2 expects the exact stub; after Phase 3, a resolved rule must have no `"placeholder"` values, use the owning stage's `-` on inputs/outputs, and carry root bindings. A remaining stub must map to a reported unresolved connector. +Confirm target task's `entryConditions[]` length equals the number of task-entry T-tasks tasks.md wrote for this task. Each entry carries `id` (prefix `c`), non-empty `displayName` (SDD value or `Entry Rule {N}` default), and `rules` with the expected `rule` value plus any required side field. For `wait-for-connector`, Phase 2 expects the exact stub; after Phase 3, EVERY such rule must have no `"placeholder"` values, use the owning stage's `-` on inputs/outputs, and carry root bindings. **A stub surviving Phase 3 is case-fatal, not a reportable open item** — it makes the whole case non-startable and halts the build at [implementation.md § Step 12 Check 14](../../../implementation.md#step-12--end-of-phase-3-validator-pass). diff --git a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/check_connector_stub_halt.py b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/check_connector_stub_halt.py new file mode 100644 index 0000000000..f6b3c66796 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/check_connector_stub_halt.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +"""Grade the Check 14 halt: an unresolvable wait-for-connector rule must never ship silently. + +Regression guarded +------------------ +Connector-event subscriptions register at CASE START. One surviving placeholder stub +anywhere makes the ENTIRE case non-startable — the case's own start event fails and +exactly one element executes — even when the stub sits on a non-required secondary +stage nothing routes to. `uip maestro case validate` reports `Valid` throughout, so +validate cannot be the gate. + +Measured on uip 1.198 (2026-08-12), same case, one field changed: + placeholder stub -> start event Failed, 1 element, validate Valid + resolved connector-> start event Completed, 25 elements, validate Valid + +Before the fix, the skill emitted the stub, logged it as an ordinary Open Item, and +reported the build complete. This checker fails exactly that outcome. + +Pass conditions (either branch is a legitimate resolution) +--------------------------------------------------------- +A. No stub survives -> the agent resolved the connector or removed the rule. + Additionally require structural coherence, so "removal" cannot mean gutting the + file into something incoherent. +B. A stub survives -> the agent MUST have halted: tasks/build-issues.md carries the + literal BLOCKED marker line from implementation.md Step 12 Check 14. + +Fail +---- +A stub survives with no BLOCKED marker -> a dead case was emitted as if complete. +That is the pre-fix behaviour and the whole point of this test. +""" + +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + +# implementation.md § Step 12 Check 14 mandates this literal line in build-issues.md. +BLOCKED_MARKER = re.compile( + r"BLOCKED:\s*case is not startable\s*[—-]\s*\d+\s+unresolved wait-for-connector rule\(s\)" + r"\s+still carry the placeholder stub\.", + re.IGNORECASE, +) + +PLACEHOLDER = "placeholder" + + +def find_one(root: Path, name: str) -> Path | None: + hits = [ + p + for p in root.rglob(name) + if ".venv" not in p.parts and "node_modules" not in p.parts + ] + return hits[0] if hits else None + + +def context_is_stub(rule: dict) -> bool: + """True when the rule still carries the Phase 2 placeholder context.""" + ctx = (rule.get("uipath") or {}).get("context") or [] + if not isinstance(ctx, list): + return False + by_name = { + e.get("name"): e.get("value") + for e in ctx + if isinstance(e, dict) + } + return ( + by_name.get("connectorKey") == PLACEHOLDER + or by_name.get("operation") == PLACEHOLDER + ) + + +def walk_conditions(case: dict): + """Yield (scope, owner_label, condition, rule) across all four condition scopes.""" + for cond in (case.get("metadata") or {}).get("caseExitRules") or []: + for grp in cond.get("rules") or []: + for rule in grp: + yield "case-exit", "root", cond, rule + + for node in case.get("nodes") or []: + if node.get("type") != "case-management:Stage": + continue + data = node.get("data") or {} + label = data.get("label") or node.get("id") + for scope_key, scope in ( + ("entryConditions", "stage-entry"), + ("exitConditions", "stage-exit"), + ): + for cond in data.get(scope_key) or []: + for grp in cond.get("rules") or []: + for rule in grp: + yield scope, label, cond, rule + for lane in data.get("tasks") or []: + for task in lane: + for cond in task.get("entryConditions") or []: + for grp in cond.get("rules") or []: + for rule in grp: + yield ( + "task-entry", + f"{label} / {task.get('displayName')}", + cond, + rule, + ) + + +def structural_problems(case: dict) -> list[str]: + """Catch a 'removal' that left the plan incoherent.""" + problems: list[str] = [] + stage_ids = { + n.get("id") + for n in case.get("nodes") or [] + if n.get("type") == "case-management:Stage" + } + + for scope, owner, cond, rule in walk_conditions(case): + where = f"{scope} on {owner}, condition {cond.get('id')!r}" + if not rule.get("rule"): + problems.append(f"{where}: rule {rule.get('id')!r} has no rule type") + target = rule.get("selectedStageId") + if target and target not in stage_ids: + problems.append(f"{where}: selectedStageId {target!r} points at a removed stage") + + for node in case.get("nodes") or []: + if node.get("type") != "case-management:Stage": + continue + data = node.get("data") or {} + for scope_key in ("entryConditions", "exitConditions"): + for cond in data.get(scope_key) or []: + groups = cond.get("rules") or [] + if not groups or not any(groups): + problems.append( + f"stage {data.get('label')!r} {scope_key}: condition " + f"{cond.get('id')!r} has an empty rules array" + ) + return problems + + +def main() -> int: + root = Path.cwd() + + caseplan_path = find_one(root, "caseplan.json") + if caseplan_path is None: + print( + "FAIL: no caseplan.json anywhere under the sandbox.\n" + " Rule 8 requires the case STRUCTURE to land even when a resource is\n" + " unresolved (placeholder path) — halting before Phase 2 is not the\n" + " expected behaviour for an unresolvable connector." + ) + return 1 + + try: + case = json.loads(caseplan_path.read_text()) + except (OSError, json.JSONDecodeError) as exc: + print(f"FAIL: caseplan.json is unreadable/unparseable: {exc}") + return 1 + + stubs = [ + (scope, owner, cond.get("displayName"), rule.get("id")) + for scope, owner, cond, rule in walk_conditions(case) + if rule.get("rule") == "wait-for-connector" and context_is_stub(rule) + ] + + issues_path = find_one(root, "build-issues.md") + issues_text = "" + if issues_path is not None: + try: + issues_text = issues_path.read_text() + except OSError: + issues_text = "" + + print(f"caseplan: {caseplan_path.relative_to(root)}") + print(f"surviving placeholder connector-rule stubs: {len(stubs)}") + for scope, owner, display, rule_id in stubs: + print(f" - [{scope}] {owner} :: {display!r} rule={rule_id}") + + if stubs: + if BLOCKED_MARKER.search(issues_text): + print( + "\nPASS (branch B): a stub survives AND the build halted with the " + "Check 14 BLOCKED marker in build-issues.md." + ) + return 0 + where = ( + issues_path.relative_to(root) if issues_path is not None else "build-issues.md (missing)" + ) + print( + f"\nFAIL: {len(stubs)} placeholder connector-rule stub(s) survived, but " + f"{where} carries no Check 14 BLOCKED marker.\n" + " Expected a literal line matching:\n" + " BLOCKED: case is not startable — unresolved wait-for-connector " + "rule(s) still carry the placeholder stub.\n" + " Without the halt this case is 100% non-startable (start event fails, " + "1 element executes)\n" + " while `uip maestro case validate` still reports Valid — emitted as if " + "it were complete.\n" + " See implementation.md § Step 12 Check 14." + ) + return 1 + + problems = structural_problems(case) + if problems: + print("\nFAIL: no stub survives, but the plan was left structurally incoherent:") + for p in problems: + print(f" - {p}") + return 1 + + print( + "\nPASS (branch A): no placeholder connector-rule stub survives and the " + "remaining conditions are coherent." + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/fixtures/sdd.md b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/fixtures/sdd.md new file mode 100644 index 0000000000..09e8b76299 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/fixtures/sdd.md @@ -0,0 +1,282 @@ +# SDD — ClaimIntakeWatch + +> Case Definition Blueprint for ClaimIntakeWatch. + +--- + +## Document History + +| Version | Date | Author | Change | +|---|---|---|---| +| 1.0 | 2026-08-12 | uipath-planner | Initial case design | + +--- + + + +## Planner Handoff + +| Field | Value | +|---|---| +| **Status** | ready | +| **Execution autonomy** | autonomous | +| **Delivery model** | cloud | +| **SDD scope** | single-product | +| **Project list section** | §2: Stages & Tasks + §4: Integrations | +| **Tasks file** | `claim-intake-watch-tasks.md` | +| **Generated by** | uipath-planner | +| **Generation date** | 2026-08-12 | +| **Template validation** | passed | + +--- + +## Table of Contents + +1. [Case Definition](#section-1-case-definition) — Metadata, Triggers, Exit Conditions, Variables +2. [Stages & Tasks](#section-2-stages--tasks) + - [Stage 1: Intake Hold](#stage-1-intake-hold) — 1 task + - [Stage 2: Withdrawal Received](#stage-2-withdrawal-received) — 1 task +3. [Personas & App Views](#section-3-personas--app-views) — 1 Persona +4. [Integrations](#section-4-integrations) — IS Connectors + +--- + +## Recommended Scope + +**Recommendation:** SINGLE_PRODUCT — Case Management +**Delivery model:** cloud +**Blocked by platform:** none +**Need profile:** A claim sits in an intake hold. Separately, the claimant may withdraw the +claim at any point, which arrives as an inbound event from the Acme Claims intake webhook +connector and diverts the case into a withdrawal lane. + +--- + +## Section 1: Case Definition + +**Purpose:** Top-level case configuration — identity, triggers, exit conditions, and the complete variable inventory. + +### Case Metadata + +| Property | Value | +|----------|-------| +| Case Name | ClaimIntakeWatch | +| Case Description | Holds a claim briefly during intake, while watching for an inbound withdrawal event from the Acme Claims intake webhook connector. | +| Case Identifier | Type: constant. Constant → Prefix: CIW | +| Priority | Choiceset: Low, Medium, High — Default: Medium | +| Case-Level SLA | — | +| SLA Type | time-based | +| Case App | Disabled | +| Task-output passing | Direct | +| Case Identifier source | =metadata.ExternalId (platform-generated — default) | + +### Case Triggers + +| T# | Trigger Type | Source | Configuration | +|----|-------------|--------|---------------| +| T02 | Manual | Manual | N/A | + +### Case Exit Conditions + +| WHEN | IF | THEN | Marks Case Complete | Display Name | +|------|-----|------|---------------------|--------------| +| required-stages-completed | — | Case exited | Yes | Intake complete | + +### Case Variables + +> None. + +--- + +## Section 2: Stages & Tasks + +**Purpose:** The case plan. Every stage has entry/exit conditions, optional SLA, and complete task detail blocks. Case has no BPMN-style edges; transitions are expressed by stage entry/exit conditions. + +--- + +### Stage 1: Intake Hold + +**Type:** Stage +**Stage Kind:** primary +**Design Rationale:** The case needs one required milestone so `required-stages-completed` has something to key off. A short timer stands in for the intake dwell. +**Description:** Holds the claim briefly during intake. +**Required for Case Completion:** Yes + +#### Stage Entry Conditions + +| WHEN | IF | Interrupting | Display Name | +|------|-----|-------------|--------------| +| case-entered | — | No | Case started | + +#### Stage Exit Conditions + +| WHEN | IF | Exit Type | Marks Stage Complete | Display Name | +|------|-----|-----------|---------------------|--------------| +| required-tasks-completed | — | exit-only | Yes | Intake hold done | + +#### Tasks + +| # | Task Name | Type | Activation Mode | Starts When | Required | Run Only Once | Persona | SLA | +|---|-----------|------|-----------------|-------------|----------|---------------|---------|-----| +| 1 | Hold Briefly | wait-for-timer | parallel | stage enters | Yes | No | system | — | + +##### Task 1.1: Hold Briefly + +**Type:** wait-for-timer +**Activation Mode:** parallel +**Design Rationale:** A deterministic, resource-free dwell — it needs no tenant registry entry, keeping the case's only unresolved resource the withdrawal connector. +**Description:** Waits a few seconds to represent the intake dwell. + +**Entry Condition:** + +| WHEN | IF | Display Name | +|------|-----|--------------| +| current-stage-entered | — | Stage entered | + +**Task envelope** + +| Required | Run Only Once | Skip Condition | +|----------|---------------|----------------| +| Yes | No | — | + +###### Timer Task Detail (type: `wait-for-timer`) + +**Duration:** 5 seconds + +--- + +### Stage 2: Withdrawal Received + +**Type:** Stage +**Stage Kind:** secondary +**Design Rationale:** Withdrawal can arrive at any point and is not part of the happy path, so it is a non-required secondary lane rather than a step in the main flow. +**Description:** Handles a claim the claimant withdrew mid-intake. +**Required for Case Completion:** No + +#### Stage Entry Conditions + +| WHEN | IF | Interrupting | Display Name | +|------|-----|-------------|--------------| +| wait-for-connector | — | Yes | Withdrawal received | + +**Connector Detail for the entry rule:** + +**Connector:** Acme Claims Intake · **Connector Key:** acme-claims-intake-webhook +**Connection:** Acme Claims Intake (production) +**Trigger / Event:** Claim Withdrawn +**Auth Method:** Connection-managed + +#### Stage Exit Conditions + +| WHEN | IF | Exit Type | Marks Stage Complete | Display Name | +|------|-----|-----------|---------------------|--------------| +| required-tasks-completed | — | exit-only | Yes | Withdrawal recorded | + +#### Tasks + +| # | Task Name | Type | Activation Mode | Starts When | Required | Run Only Once | Persona | SLA | +|---|-----------|------|-----------------|-------------|----------|---------------|---------|-----| +| 1 | Record Withdrawal | wait-for-timer | parallel | stage enters | Yes | No | system | — | + +##### Task 1.1: Record Withdrawal + +**Type:** wait-for-timer +**Activation Mode:** parallel +**Design Rationale:** Stands in for the close-out bookkeeping so the lane has a required task for its exit rule. +**Description:** Brief dwell representing withdrawal close-out. + +**Entry Condition:** + +| WHEN | IF | Display Name | +|------|-----|--------------| +| current-stage-entered | — | Stage entered | + +**Task envelope** + +| Required | Run Only Once | Skip Condition | +|----------|---------------|----------------| +| Yes | No | — | + +###### Timer Task Detail (type: `wait-for-timer`) + +**Duration:** 5 seconds + +--- + +## Section 3: Personas & App Views + +**Purpose:** Who interacts with the case and through what interfaces. + +### Personas + +| Persona | Stage Scope | Permissions | Description | +|---------|-------------|-------------|-------------| +| Claims Handler | All | View, Act | Starts the case and reviews the outcome. | + +### Process App Views + +| App | View | Persona | Purpose | Key Components | +|-----|------|---------|---------|----------------| +| — | — | — | Case App is disabled. | — | + +--- + +## Section 4: Integrations + +**Purpose:** Complete inventory of intended or deployed resources and external systems the case binds. + +### Integration Service Connectors + +| Connector | Connector Key | System | Connection (ID) | Auth Method | Operations Used | Used By Tasks | +|-----------|---------------|--------|-----------------|-------------|-----------------|---------------| +| Acme Claims Intake | acme-claims-intake-webhook | Acme Claims intake webhook | Acme Claims Intake (production) | Connection-managed | Claim Withdrawn (event) | Stage 2 entry rule | + +### API Workflows + +> None. + +### Agents + +> None. + +### Processes & RPA + +> None. + +### Child Cases + +> None. + +### External Agents + +> None. + +--- + +## Non-Functional Requirements + +| Dimension | Requirement / Design decision | +|---|---| +| Security | No PII in this cut. | +| Performance | Single short dwell per lane. | +| Scalability | Not applicable. | +| Availability / Resilience | No retry path modeled. | +| Logging & Monitoring | Standard case instance telemetry. | +| Compliance | None applicable. | + +## Testing Strategy + +| Scenario | Setup | Expected Outcome | +|---|---|---| +| Happy path | Start the case manually | Intake Hold enters, Hold Briefly completes, case completes | +| Withdrawal | Inbound Claim Withdrawn event | Withdrawal Received lane interrupts and records the withdrawal | + +--- + +## Next Steps + +To build the case, load `uipath-maestro-case` with this file as the source `sdd.md` and preserve the Section 1-4 content unchanged. + +--- + +**End of Case Definition Blueprint.** diff --git a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml new file mode 100644 index 0000000000..8e83a96673 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml @@ -0,0 +1,125 @@ +# --- No tenant fixtures required ------------------------------------------------------ +# The connector named by the SDD (`acme-claims-intake-webhook`) is DELIBERATELY absent +# from every tenant. The test asserts what the skill does when a wait-for-connector rule +# cannot be resolved — it must NOT ship the case. Do not "fix" this by provisioning an +# Acme connector; an unresolvable connector IS the fixture. +# -------------------------------------------------------------------------------------- +task_id: skill-case-unresolved-connector-rule-halt +description: > + CASE-FATAL PLACEHOLDER GUARD — an approved sdd.md puts a wait-for-connector entry rule + on a non-required secondary stage, bound to a connector that exists on no tenant. Phase 2 + legitimately writes the placeholder stub; Phase 3 Step 10.5 cannot upgrade it. Step 12 + Check 14 must then HALT: a surviving stub registers no subscription at case start, which + makes the ENTIRE case non-startable (start event fails, 1 element executes) even though + the stub sits on a stage nothing routes to, and `uip maestro case validate` reports Valid + throughout. Regression guarded: the skill used to emit the stub, log it as an ordinary + Open Item, and report the build complete — shipping a 100% dead case. Measured on + uip 1.198 (2026-08-12): placeholder stub -> 1 element / Faulted; same rule with a + resolved connector -> 25 elements. +tags: + - uipath-maestro-case + - integration + - lifecycle:generate + - mode:build + - shape:multi-node + - resource + - feature:trigger + - feature:registry + +agent: + type: claude-code + permission_mode: acceptEdits + allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep", "AskUserQuestion", "TodoWrite"] + +run_limits: + task_timeout: 3600 + max_turns: 140 + turn_timeout: 1800 + +sandbox: + template_sources: + - type: template_dir + path: fixtures + +initial_prompt: | + This approved `sdd.md` is authoritative. Build the case from it: solution, project, + and `caseplan.json`, through validation. + + Do not publish to Studio Web, do not publish to Orchestrator, and do not run a debug + session — stop once the case is built and validated, or sooner if you determine the + case should not be built at all. + + The `uip` CLI is already available and authenticated. Use the SDD as written: do not + redo the design and do not substitute a different connector for the one it names. + Before starting, load the uipath-maestro-case skill and follow its workflow. + +success_criteria: + - type: run_command + description: "Rule 8 held — the case structure landed on disk despite the unresolved connector" + command: "find . -name caseplan.json -not -path '*/.venv/*' | grep -q ." + timeout: 10 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: > + PRIMARY. Either no placeholder connector-rule stub survives (agent removed or + resolved the rule, leaving a coherent plan), or one survives AND the build halted + with the Check 14 BLOCKED marker. A surviving stub with no marker is the regression. + command: "python3 $TASK_DIR/check_connector_stub_halt.py" + timeout: 30 + expected_exit_code: 0 + weight: 5.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "A case that cannot start was never executed" + tool_name: "Bash" + command_pattern: 'uip\s+maestro\s+case\s+debug' + weight: 3.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "A case that cannot start was never shipped to the tenant" + tool_name: "Bash" + command_pattern: 'uip\s+solution\s+(upload|publish)' + weight: 3.0 + pass_threshold: 1.0 + + - type: run_command + description: "The build recorded its issues rather than exiting silently" + command: "find . -name build-issues.md -not -path '*/.venv/*' | grep -q ." + timeout: 10 + expected_exit_code: 0 + weight: 1.0 + pass_threshold: 1.0 + + - type: run_command + description: > + The unresolvable connector is named in the build record, so a reader can act on it + (advisory — the PRIMARY criterion grades the halt itself). + command: "grep -rqi 'acme-claims-intake-webhook\\|Acme Claims Intake' --include=build-issues.md ." + timeout: 10 + expected_exit_code: 0 + weight: 1.0 + pass_threshold: 0 + +simulation: + enabled: true + persona: | + You are a claims solution designer. The SDD is yours and it is correct as written: + withdrawal genuinely does arrive from the Acme Claims intake webhook. What you do NOT + have is that connector on this tenant — procurement has not set it up yet, and you + cannot supply a connection, a connector key, or a substitute connector. + goal: | + Find out what the build does with a connector you cannot provide. You would rather be + told the case is not startable than be handed something that looks finished and dies + on the first run. + constraints: + - "You cannot provide the Acme connector, a connection id, or any credentials for it — it does not exist on this tenant." + - "Do not accept a substitute connector, and do not agree to redesign the withdrawal lane around a different trigger." + - "If asked how to proceed on the unresolved connector rule, choose to stop the build rather than emit something that cannot start." + - "Do not authorise publish, upload, or a debug run." + - "Keep replies short — one sentence or a single option pick." + max_turns: 12 From 189c832c1cff5020bbf95a7f49a6dc8f364ad5f0 Mon Sep 17 00:00:00 2001 From: roberts-cliff Date: Thu, 13 Aug 2026 08:26:49 -0600 Subject: [PATCH 2/4] test(maestro-case): unit-test the Check 14 grader against synthetic stubs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The grader's only evidence was a manual run against a captured build artifact on one machine — not reproducible, not in CI, and it rots the moment that sample changes. Replace it with 16 pytest cases built on minimal synthetic stubs: root + one stage + one placeholder rule, the smallest shape that reproduces the fault. No tenant, no captured artifact, runs anywhere. Covers the regression directly (surviving stub with no BLOCKED marker must fail), both pass branches, all four condition scopes, a partial stub, and the "removal left the plan incoherent" guards. The load-bearing case is test_open_item_mention_is_not_a_halt: the pre-fix skill DID log the stub, as a routine Open Item. A grader that accepts any mention of "placeholder" would pass the exact output this PR exists to reject. Mutation-tested — each mutation is caught by its designated test: grader always returns 0 -> 15 of 16 fail stage-exit scope not scanned -> scope[stage_exit] fails any "placeholder" is a halt -> open-item test fails Also adds the skill_triggered criterion for parity with sibling connector tasks (PR review, Low). Co-Authored-By: Claude Opus 5 --- .../test_connector_stub_halt.py | 292 ++++++++++++++++++ .../unresolved_connector_rule_halt.yaml | 10 +- 2 files changed, 301 insertions(+), 1 deletion(-) create mode 100644 tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/test_connector_stub_halt.py diff --git a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/test_connector_stub_halt.py b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/test_connector_stub_halt.py new file mode 100644 index 0000000000..c81e9da443 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/test_connector_stub_halt.py @@ -0,0 +1,292 @@ +"""Unit tests for check_connector_stub_halt.py — the Check 14 grader. + +Run with ``pytest`` from any directory; CI runs them via the +``maestro-case checker unit tests`` job (``pytest tests/tasks/uipath-maestro-case/``). + +Every fixture here is a MINIMAL SYNTHETIC STUB built inline — the smallest +caseplan that carries the fatal shape (root + one stage + one placeholder +`wait-for-connector` rule). No real or captured build artifact is involved, so +these run anywhere, need no tenant, and cannot rot when a sample case changes. + +What the grader must get right +------------------------------ +A surviving placeholder connector-rule stub makes the ENTIRE case non-startable: +subscriptions register at case start, so one stub anywhere kills the case's own +start event even when it sits on a stage nothing routes to. `uip maestro case +validate` reports `Valid` either way, so the grader is the only signal. + + branch A no stub survives, plan coherent -> pass + branch B stub survives AND BLOCKED marker -> pass (the build halted) + FAIL stub survives, no BLOCKED marker -> the pre-fix regression +""" + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +CHECKER = Path(__file__).with_name("check_connector_stub_halt.py") + +BLOCKED_LINE = ( + "BLOCKED: case is not startable — 1 unresolved wait-for-connector rule(s) " + "still carry the placeholder stub." +) + +STUB_UIPATH = { + "serviceType": "Intsvc.WaitForEvent", + "context": [ + {"name": "connectorKey", "value": "placeholder", "type": "string"}, + {"name": "operation", "value": "placeholder", "type": "string"}, + ], + "inputs": [], + "outputs": [], + "bindings": [], +} + +RESOLVED_UIPATH = { + "serviceType": "Intsvc.WaitForEvent", + "context": [ + {"name": "connectorKey", "value": "uipath-mock-element", "type": "string"}, + {"name": "connection", "value": "=bindings.bA1B2C3D4", "type": "string"}, + {"name": "objectName", "value": "tokens", "type": "string"}, + {"name": "operation", "value": "TOKEN_CREATED", "type": "string"}, + ], + "inputs": [], + "outputs": [], + "bindings": [], +} + + +def connector_rule(uipath): + return {"id": "Rule_aaaaaa", "rule": "wait-for-connector", "uipath": uipath} + + +def caseplan(*, stage_entry=None, stage_exit=None, task_entry=None, case_exit=None): + """Smallest caseplan that can carry a rule in any of the four scopes.""" + task = {"id": "tAAAAAAAA", "type": "wait-for-timer", "displayName": "Hold"} + if task_entry is not None: + task["entryConditions"] = [ + {"id": "cAAAAAAAA", "displayName": "Task entry", "rules": [[task_entry]]} + ] + + stage = { + "id": "Stage_aaaaaa", + "type": "case-management:Stage", + "data": { + "label": "Watch", + "isRequired": False, + "stageType": "secondary", + "tasks": [[task]], + }, + } + if stage_entry is not None: + stage["data"]["entryConditions"] = [ + {"id": "Condition_aaaaaa", "displayName": "Entry", "rules": [[stage_entry]]} + ] + if stage_exit is not None: + stage["data"]["exitConditions"] = [ + { + "id": "Condition_bbbbbb", + "displayName": "Exit", + "type": "exit-only", + "marksStageComplete": True, + "rules": [[stage_exit]], + } + ] + + metadata = {"caseIdentifier": "STB"} + if case_exit is not None: + metadata["caseExitRules"] = [ + { + "id": "Condition_cccccc", + "displayName": "Done", + "marksCaseComplete": True, + "rules": [[case_exit]], + } + ] + + return { + "id": "case-Stub000001", + "version": "27.0.0", + "name": "Stub", + "metadata": metadata, + "bindings": [], + "variables": {"inputs": [], "outputs": [], "inputOutputs": []}, + "nodes": [stage], + "edges": [], + "layout": {}, + } + + +def run_checker(tmp_path, case=None, issues=None, nested="Sol/Proj"): + """Lay the stub out on disk the way a real build would, then grade it.""" + if case is not None: + project = tmp_path / nested + project.mkdir(parents=True, exist_ok=True) + (project / "caseplan.json").write_text(json.dumps(case, indent=2)) + if issues is not None: + tasks_dir = tmp_path / "tasks" + tasks_dir.mkdir(parents=True, exist_ok=True) + (tasks_dir / "build-issues.md").write_text(issues) + + return subprocess.run( + [sys.executable, str(CHECKER)], + cwd=tmp_path, + capture_output=True, + text=True, + ) + + +# --- the regression this test exists for -------------------------------------------- + +def test_surviving_stub_without_marker_fails(tmp_path): + """Pre-fix behaviour: emit the dead case, report it complete. Must FAIL.""" + res = run_checker(tmp_path, case=caseplan(stage_entry=connector_rule(STUB_UIPATH))) + assert res.returncode == 1, res.stdout + assert "no Check 14 BLOCKED marker" in res.stdout + + +def test_surviving_stub_with_marker_passes(tmp_path): + """Post-fix branch B: the stub is still there, but the build halted.""" + res = run_checker( + tmp_path, + case=caseplan(stage_entry=connector_rule(STUB_UIPATH)), + issues=f"# Build Issues\n\n{BLOCKED_LINE}\n\n## Open Items for User\n", + ) + assert res.returncode == 0, res.stdout + assert "PASS (branch B)" in res.stdout + + +def test_resolved_connector_passes(tmp_path): + """Post-fix branch A: the connector resolved, so nothing is fatal.""" + res = run_checker(tmp_path, case=caseplan(stage_entry=connector_rule(RESOLVED_UIPATH))) + assert res.returncode == 0, res.stdout + assert "PASS (branch A)" in res.stdout + + +def test_rule_removed_entirely_passes(tmp_path): + """Branch A via removal: no connector rule at all.""" + case = caseplan() + case["nodes"][0]["data"]["entryConditions"] = [ + { + "id": "Condition_aaaaaa", + "displayName": "Entry", + "rules": [[{"id": "Rule_bbbbbb", "rule": "case-entered"}]], + } + ] + res = run_checker(tmp_path, case=case) + assert res.returncode == 0, res.stdout + assert "PASS (branch A)" in res.stdout + + +# --- scope coverage: a stub in ANY of the four scopes is fatal ----------------------- + +@pytest.mark.parametrize("scope", ["stage_entry", "stage_exit", "task_entry", "case_exit"]) +def test_stub_detected_in_every_condition_scope(tmp_path, scope): + res = run_checker(tmp_path, case=caseplan(**{scope: connector_rule(STUB_UIPATH)})) + assert res.returncode == 1, f"{scope} stub went undetected:\n{res.stdout}" + assert "surviving placeholder connector-rule stubs: 1" in res.stdout + + +def test_partial_stub_counts_as_a_stub(tmp_path): + """Only `operation` left as placeholder — still not runnable, still fatal.""" + half = dict(RESOLVED_UIPATH) + half["context"] = [ + {"name": "connectorKey", "value": "uipath-mock-element", "type": "string"}, + {"name": "operation", "value": "placeholder", "type": "string"}, + ] + res = run_checker(tmp_path, case=caseplan(stage_entry=connector_rule(half))) + assert res.returncode == 1, res.stdout + + +# --- the marker must be the real one, not any mention of the problem ---------------- + +def test_open_item_mention_is_not_a_halt(tmp_path): + """The pre-fix skill DID log the stub — as a routine Open Item. Not a halt.""" + res = run_checker( + tmp_path, + case=caseplan(stage_entry=connector_rule(STUB_UIPATH)), + issues=( + "# Build Issues\n\n## Open Items for User\n\n" + "- The withdrawal lane can't fire; replace the placeholder connector " + "values before debug / publish-to-run.\n" + ), + ) + assert res.returncode == 1, ( + "an Open Item mentioning the placeholder must NOT satisfy Check 14 — " + "that is exactly the pre-fix output:\n" + res.stdout + ) + + +def test_missing_build_issues_file_is_not_a_halt(tmp_path): + res = run_checker(tmp_path, case=caseplan(stage_entry=connector_rule(STUB_UIPATH))) + assert res.returncode == 1 + assert "build-issues.md (missing)" in res.stdout + + +# --- removal must not mean gutting the plan ------------------------------------------ + +def test_removal_leaving_dangling_stage_reference_fails(tmp_path): + case = caseplan() + case["nodes"][0]["data"]["entryConditions"] = [ + { + "id": "Condition_aaaaaa", + "displayName": "Entry", + "rules": [ + [ + { + "id": "Rule_bbbbbb", + "rule": "selected-stage-exited", + "selectedStageId": "Stage_deleted", + } + ] + ], + } + ] + res = run_checker(tmp_path, case=case) + assert res.returncode == 1, res.stdout + assert "points at a removed stage" in res.stdout + + +def test_removal_leaving_empty_rules_array_fails(tmp_path): + case = caseplan() + case["nodes"][0]["data"]["entryConditions"] = [ + {"id": "Condition_aaaaaa", "displayName": "Entry", "rules": []} + ] + res = run_checker(tmp_path, case=case) + assert res.returncode == 1, res.stdout + assert "empty rules array" in res.stdout + + +# --- environment guards --------------------------------------------------------------- + +def test_no_caseplan_fails(tmp_path): + res = run_checker(tmp_path) + assert res.returncode == 1 + assert "no caseplan.json" in res.stdout + + +def test_unparseable_caseplan_fails(tmp_path): + project = tmp_path / "Sol" / "Proj" + project.mkdir(parents=True) + (project / "caseplan.json").write_text("{ not json") + res = subprocess.run( + [sys.executable, str(CHECKER)], cwd=tmp_path, capture_output=True, text=True + ) + assert res.returncode == 1 + assert "unreadable/unparseable" in res.stdout + + +def test_venv_copies_are_ignored(tmp_path): + """A caseplan inside .venv must not be mistaken for the build output.""" + venv_project = tmp_path / ".venv" / "share" / "sample" + venv_project.mkdir(parents=True) + (venv_project / "caseplan.json").write_text( + json.dumps(caseplan(stage_entry=connector_rule(STUB_UIPATH))) + ) + res = run_checker(tmp_path, case=caseplan(stage_entry=connector_rule(RESOLVED_UIPATH))) + assert res.returncode == 0, res.stdout + assert ".venv" not in res.stdout diff --git a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml index 8e83a96673..cb07da4e24 100644 --- a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml +++ b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml @@ -15,7 +15,8 @@ description: > throughout. Regression guarded: the skill used to emit the stub, log it as an ordinary Open Item, and report the build complete — shipping a 100% dead case. Measured on uip 1.198 (2026-08-12): placeholder stub -> 1 element / Faulted; same rule with a - resolved connector -> 25 elements. + resolved connector -> 25 elements. The grader itself is unit-tested against minimal + synthetic stubs in test_connector_stub_halt.py (no tenant, no captured artifact). tags: - uipath-maestro-case - integration @@ -87,6 +88,13 @@ success_criteria: weight: 3.0 pass_threshold: 1.0 + - type: skill_triggered + description: "Agent invoked the uipath-maestro-case skill" + skill_name: "uipath-maestro-case" + expected_skill: "uipath-maestro-case" + weight: 1.0 + pass_threshold: 1.0 + - type: run_command description: "The build recorded its issues rather than exiting silently" command: "find . -name build-issues.md -not -path '*/.venv/*' | grep -q ." From d7b2c67b4f1a296142ed4286c4d7beeb1c487292 Mon Sep 17 00:00:00 2001 From: roberts-cliff Date: Thu, 13 Aug 2026 09:09:49 -0600 Subject: [PATCH 3/4] =?UTF-8?q?fix(maestro-case):=20Check=2014=20is=20the?= =?UTF-8?q?=20halt=20point=20=E2=80=94=20"must=20not=20ship"=20is=20not=20?= =?UTF-8?q?"must=20not=20build"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught by running the coder-eval task end-to-end; the unit tests cannot see it, because they grade the checker rather than the skill's behaviour. The first Check 14 wording said a surviving stub "may NOT ship" without saying WHERE the refusal happens. The agent resolved that ambiguity by refusing to build at all: it did a thorough Phase 1 (forced registry pull, searched every cached index, `is connectors list`, `is connections list`, correctly separated "absent from the catalog" from "present but unprovisioned"), concluded the case could never start, and stopped without creating a solution, a caseplan, or a build-issues.md. Task scored 0.452 — no artifact for the two structural criteria to grade. Defensible reasoning, wrong outcome. Check 14's remediation (b) — remove the rule and ship the working remainder — is usually the only option that unblocks the author without waiting on the connector, and it only exists once the case has been built. Bailing at discovery silently deletes it and hands back prose instead of a reviewable artifact. The run's transcript shows exactly that: three options offered, none of them "drop the lane and ship the rest". Clarifies in four places that discovery does not abort the run: emit the Rule 8 stub, build the structure, refuse at Check 14. Re-ran after the fix: SUCCESS, 7/7 criteria, weighted score 1.0, branch B (structure lands, stub survives, BLOCKED marker written, no debug, no publish), and the closing report explicitly does not call the build complete. Co-Authored-By: Claude Opus 5 --- skills/uipath-maestro-case/SKILL.md | 2 +- skills/uipath-maestro-case/references/connector-trigger-impl.md | 2 ++ skills/uipath-maestro-case/references/implementation.md | 2 ++ skills/uipath-maestro-case/references/placeholder-tasks.md | 2 +- 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/skills/uipath-maestro-case/SKILL.md b/skills/uipath-maestro-case/SKILL.md index f16278278a..3c099e1e9e 100644 --- a/skills/uipath-maestro-case/SKILL.md +++ b/skills/uipath-maestro-case/SKILL.md @@ -35,7 +35,7 @@ When `sdd.md` is absent, the case design belongs to **`uipath-planner` — the s 5. **Follow plugin per node type.** Open matching `planning.md` during planning + `impl-json.md` during execution. Never guess JSON shapes from memory. 6. **`tasks.md` declarative and lossless only.** No shell commands inside. Field names use plain identifiers (e.g., `type:`, `displayName:`, `lane:`), not CLI flag syntax. One T-entry per sdd.md declaration — every stage, task, trigger, condition, SLA rule, **variable, and argument** gets own T-number, even when value looks like default (`current-stage-entered`, `case-entered`, `exit-only`, `is-interrupting: false`, `runOnlyOnce: true`, `marks-stage-complete: true`). Never group, never silently omit. **An explicit stage/task entry or exit rule in a supplied or approved SDD is authoritative: planning and implementation preserve that exact rule and its selectors, even when a different rule would normally be inferred from task proximity or list order.** Preserve every stage/task/SLA `Design Rationale` and condition routing/activation rationale as `rationale:` on the matching T-entry; rationale is reviewer/audit context and never changes the executable JSON shape. Preserve every SDD Inputs row with its declared binding mode and value. A JSON object literal stays literal through both handoffs: record the exact JSON in `tasks.md`, then write either the native object or its JSON-encoded string to `input.value`; never add `=js:` or `=jsonString:` unless the SDD itself explicitly uses that prefix. Project every task/rule Outputs table row through the common grammar in [`plugins/variables/io-binding/planning.md`](references/plugins/variables/io-binding/planning.md#sdd-outputs-table-to-tasksmd-projection-mandatory), then preserve each resulting `outputs:` item **with its operator and both operands unchanged**. SDD Outputs rows require `->` or `=`; a bare `tasks.md` output is generated only from resolved-schema discovery and is never authored as an SDD row. SDD table placeholders such as a `—` Field are not operands and never appear in `tasks.md`. In particular, `greeting -> greeting` is NOT equivalent to schema-discovered bare `greeting`: the former extracts into the predeclared case variable and requires `originalVar`; the latter auto-mints a task-local output. Never simplify an equal-name `->` row. **When an sdd.md row's format is unrecognized, ambiguous, or cannot be categorized — invoke AskUserQuestion before skipping. Silent omission is forbidden.** Always regenerate from scratch (greenfield/planning only — brownfield targeted edits mutate in place and preserve IDs; see [references/brownfield.md](references/brownfield.md)). **Every §4.6 task T-entry carries its own `activation-mode:` and `entry-rule:` lines — a separate §4.7 `rule-type:` entry does not satisfy this.** **Every task T-entry heading quotes the task's display name** in the exact form `## T: Add task "" to ""` (e.g. `## T08: Add wait-for-timer task "First Step" to "Process"`) — an unquoted or reworded heading (e.g. `## T08: Task First Step`) breaks plan addressability and fails plan validators even when `caseplan.json` itself is correct. See [`references/planning.md` §4.0](references/planning.md) and the [Plan-shape gate](references/planning.md#step-5--finalize-tasksmd-auto-proceed-to-phase-2). 7. **`tasks.md` gate — auto-approved by default, opt-in stop.** Phase 1 auto-proceeds into Phase 2 Prototyping with no AskUserQuestion sign-off; treat the plan as approved. **Stop after `tasks.md` only when the request explicitly asked for a plan-only / review-first run** (e.g. "just the plan", "Phase 1 only", "stop after tasks.md for review", "don't build the case yet") — then report the plan and do NOT proceed to Phase 2. Re-read `tasks.md` before executing. -8. **Unresolved resource → placeholder, never fabricate IDs.** Keep `` markers in `tasks.md`. Placeholder **task**: node with `type` + `displayName` + structural fields, `data: {}`; conditions still reference the TaskId. Placeholder **event trigger**: node with render fields + `data.inputs: { serviceType: "Intsvc.EventTrigger" }` only (no other `data.inputs` keys); `entry-points.json` entry appended. No trigger-edge is created (Rule 20). **Exception — a connector-bound condition-rule stub may NOT survive the build.** Placeholder tasks and triggers are shippable build state; a `wait-for-connector` rule still carrying `connectorKey`/`operation` = `"placeholder"` after Phase 3 is **case-fatal** — subscriptions register at case start, so one stub anywhere makes the ENTIRE case non-startable (start event fails, 1 element executed) even when it sits on a secondary stage nothing routes to, and `validate` reports `Valid` throughout. Step 12 **Check 14** halts the build on any survivor with no best-effort escape. See [references/placeholder-tasks.md](references/placeholder-tasks.md), [references/connector-trigger-impl.md § Placeholder fallback](references/connector-trigger-impl.md#placeholder-fallback), and [references/plugins/triggers/event/impl-json.md § Placeholder fallback](references/plugins/triggers/event/impl-json.md). +8. **Unresolved resource → placeholder, never fabricate IDs.** Keep `` markers in `tasks.md`. Placeholder **task**: node with `type` + `displayName` + structural fields, `data: {}`; conditions still reference the TaskId. Placeholder **event trigger**: node with render fields + `data.inputs: { serviceType: "Intsvc.EventTrigger" }` only (no other `data.inputs` keys); `entry-points.json` entry appended. No trigger-edge is created (Rule 20). **Exception — a connector-bound condition-rule stub may NOT survive the build.** Placeholder tasks and triggers are shippable build state; a `wait-for-connector` rule still carrying `connectorKey`/`operation` = `"placeholder"` after Phase 3 is **case-fatal** — subscriptions register at case start, so one stub anywhere makes the ENTIRE case non-startable (start event fails, 1 element executed) even when it sits on a secondary stage nothing routes to, and `validate` reports `Valid` throughout. Step 12 **Check 14** halts the build on any survivor with no best-effort escape. **The halt is at Check 14 — never earlier.** Discovering in Phase 1 that the connector is unresolvable does NOT abort the run: emit the Rule 8 placeholder, build the structure, and carry it to Check 14. Halting at discovery looks safer but is strictly worse — it produces no `caseplan.json`, no `tasks.md`, and no `build-issues.md`, and it denies the user Check 14's remediation (b), *remove the rule and ship the rest*, which is usually the only option that unblocks them today. See [references/placeholder-tasks.md](references/placeholder-tasks.md), [references/connector-trigger-impl.md § Placeholder fallback](references/connector-trigger-impl.md#placeholder-fallback), and [references/plugins/triggers/event/impl-json.md § Placeholder fallback](references/plugins/triggers/event/impl-json.md). 9. **Persist every registry resolution to `registry-resolved.json`** — one object per task with exact keys `stage`, `task`, `taskType`, `cacheFile`, `searchQuery`, `matches`, `selected`, and `rationale` (plus resolved I/O/review metadata, and `gateDecision` ONLY when the user answered the design-time gate for the resource — defaulted deferrals carry none, keeping them eligible for the Rule 17 gate). `stage` + `task` associate the audit entry to one SDD declaration; `cacheFile` is the basename actually searched; `matches` is the full exact-name match set from the cache refreshed in Rule 3, not a summary. Use the authoritative SDD fields as the search and selection contract; record `selected` from that match set, or `null` after a genuine empty lookup. **Same-session planner design:** the lane's resolution ledger (in context from the design conversation) already carries this exact shape — persist it verbatim as the file's initial content, then verify/extend per Rule 3. 10. **Cross-task refs:** plan as `"Stage Name"."Task Name".output_name`. Resolve both whole-value `<-` and in-expression `$xref` through the common output-reference-ID algorithm in [`plugins/variables/io-binding/impl-json.md`](references/plugins/variables/io-binding/impl-json.md#output-reference-id-authoritative): use the source output's `.id`; only a custom `=` output, which intentionally has no `.id`, resolves through its verified root companion's `.id`. Never use a reassigned output's `.var` as the source reference ID — it points at the target Case variable and can differ from the collision-safe source `.id`. Discover output names via `uip maestro case spec` (connector tasks) or `uip maestro case tasks describe` (non-connector tasks) — never fabricate. **Inside** a larger `=js:` expression (composite payload, condition, SLA), use the in-expression marker `vars.$xref('Stage','Task','output')` instead — resolved at Step 11.5. See [references/bindings-and-expressions.md](references/bindings-and-expressions.md) and [`plugins/variables/io-binding/impl-json.md`](references/plugins/variables/io-binding/impl-json.md). 11. **Build-review preference decides the Phase 2 → Phase 3 boundary — captured ONCE, up front, never asked mid-build.** Capture it at journey start: greenfield-with-design-handoff folds it into the lane's Case Review Build options (`Build it — straight through` / `Build it — pause at the build preview` — Rule 1); greenfield-with-provided-SDD asks it once right after the roadmap; non-interactive runs and resumed runs with no recorded preference default to **straight-through** (no mid-build publish — Phase 5 stays the only publish point and stays gated). At the boundary, try `validate --skeleton-v2` first; fall back once to legacy `--skeleton` **only** when the parser response names `--skeleton-v2` as unknown or unsupported (typically `ErrorCode: invalid_argument` and exit 3). Exit 3 alone is not enough. A genuine validation failure means v2 ran — report it and do not hide it with a fallback. Name the selected profile in the counts summary; the gate is advisory and never halts on validation findings. Then: **straight-through** → continue into Phase 3 with no prompt, the summary line doubling as the milestone narration; **pause-at-preview** → follow the publish-for-review contract in [`references/phased-execution.md`](references/phased-execution.md) (AskUserQuestion `Publish for review` / `Skip publish and continue` / `Abort`; on publish, print `DesignerUrl` as plain text BEFORE the follow-up prompt — never only inside the question body). Hard stops that are NEVER bypassed regardless of preference: Phase 4 retry exhaustion (`Retry with fix` / `Pause for manual edit` / `Abort`), Phase 5 entry (`Publish to Studio Web` / `Skip to Debug`), Phase 6 entry (`Run debug session` / `Continue to publish`), and Phase 7 entry (`Publish to Orchestrator` / `Done`). Full contract in [`references/phased-execution.md`](references/phased-execution.md). diff --git a/skills/uipath-maestro-case/references/connector-trigger-impl.md b/skills/uipath-maestro-case/references/connector-trigger-impl.md index db2f8b2180..6a4f65095f 100644 --- a/skills/uipath-maestro-case/references/connector-trigger-impl.md +++ b/skills/uipath-maestro-case/references/connector-trigger-impl.md @@ -209,6 +209,8 @@ This stub is a **deliberate mock**, and it is safe ONLY while temporary — as P > | resolved connector (spliced `caseShape`) | Completed | 25 | `Valid` | > > Placeholder **tasks** (`data: {}`, Rule 8) are NOT affected — a case with placeholder tasks starts and runs normally. The fatal shape is specific to connector-bound **condition rules**. +> +> **"Must not ship" is not "must not build."** When planning finds the connector unresolvable, still emit this stub and build the whole case — the halt belongs at Check 14, not at discovery. Aborting early yields no `caseplan.json`, no `tasks.md`, and no `build-issues.md`, so the author gets prose instead of a reviewable artifact, and Check 14's remediation (b) — *remove the rule, ship the working remainder* — never gets offered. Build it, then refuse to ship it. A remaining stub has no real outputs, Connection/Folder bindings, IS-cache entry, or rule-specific `bindings_v2` resource. Stamp unresolved `tasks.md` entries with Rule 8 markers and log them, but do NOT downgrade the finding to a completion-report note — Check 14 halts the build. Upgrade by re-running the [§ Procedure](#procedure-phase-3). diff --git a/skills/uipath-maestro-case/references/implementation.md b/skills/uipath-maestro-case/references/implementation.md index fca6734289..1a48c8870e 100644 --- a/skills/uipath-maestro-case/references/implementation.md +++ b/skills/uipath-maestro-case/references/implementation.md @@ -304,6 +304,8 @@ After value bindings (Step 9.8), connector-rule upgrades (Step 10.5), and marker - **Check 14 — No surviving connector-rule placeholder stub (CASE-FATAL, halts)** — Scan every rule across all 4 condition scopes (stage-entry / stage-exit / task-entry, plus case-exit under `metadata.caseExitRules`) for a `wait-for-connector` rule whose `uipath.context` still carries `connectorKey: "placeholder"` / `operation: "placeholder"`. **Every survivor is a build failure**, independent of its registry entry — this check does NOT exempt `selected: null` the way Check 12 does, because an unresolved connector rule is exactly the fatal case. + **This check is the halt point — do not pre-empt it.** An unresolvable connector found during Phase 1 planning or Phase 3 Step 10.5 does NOT abort the run early. Emit the Rule 8 stub, build the full structure, and arrive here. Bailing at discovery leaves no `caseplan.json`, no `tasks.md`, and no `build-issues.md` — the author gets an explanation instead of an artifact — and it silently removes remediation (b) below, *remove the rule and ship the working remainder*, which is normally the only option that unblocks them without waiting on the connector. Build it, then refuse to ship it. + **Why it halts rather than degrading.** Connector-event subscriptions register at **case start**, before any stage is entered. One surviving stub anywhere makes the ENTIRE case non-startable — including when it sits on a non-required secondary stage that nothing routes to. The observable failure is the case's own start event failing (`finalStatus: Faulted`, **1 element executed**), which points nowhere near the offending rule. `uip maestro case validate` reports `Valid` at every step, so this check is the ONLY build-time gate. Measured evidence + the one-field control/variant table: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). **Remediation — AskUserQuestion, no best-effort option.** Name the stage, condition, and rule, then offer exactly three picks: (a) **resolve the connector now** — supply/choose the connector + connection, re-run Step 10.5 for that rule, re-scan; (b) **remove the rule** — delete it and, when it was the condition's only rule, the condition too (if that leaves a stage with no entry condition, remove the stage and every reference to it, then re-run Checks 8 and 13); (c) **abort the build** — leave `caseplan.json` on disk and stop. diff --git a/skills/uipath-maestro-case/references/placeholder-tasks.md b/skills/uipath-maestro-case/references/placeholder-tasks.md index 6543c6a6f5..875a1526df 100644 --- a/skills/uipath-maestro-case/references/placeholder-tasks.md +++ b/skills/uipath-maestro-case/references/placeholder-tasks.md @@ -79,7 +79,7 @@ Case-level event triggers (`type: "uipath.case.trigger"` with `data.inputs.servi When a `wait-for-connector` rule's connector hasn't resolved at write-time, emit the rule with a **stub `uipath`** (`serviceType` + 2 `"placeholder"` context fields: `connectorKey` + `operation`). Full recipe + skip behavior + upgrade path: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). -> **This is the one placeholder the skill may NOT ship.** Unlike a placeholder task, a surviving connector-rule stub is **case-fatal**: subscriptions register at case start, so one stub anywhere makes the entire case non-startable (start event fails, 1 element executed) while `validate` still reports `Valid`. It is legal as Phase 2 build state only. If it survives Phase 3, [Step 12 Check 14](implementation.md#step-12--end-of-phase-3-validator-pass) halts the build — it is never downgraded to an Open Item. Evidence table: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). +> **This is the one placeholder the skill may NOT ship.** Unlike a placeholder task, a surviving connector-rule stub is **case-fatal**: subscriptions register at case start, so one stub anywhere makes the entire case non-startable (start event fails, 1 element executed) while `validate` still reports `Valid`. It is legal as Phase 2 build state only. If it survives Phase 3, [Step 12 Check 14](implementation.md#step-12--end-of-phase-3-validator-pass) halts the build — it is never downgraded to an Open Item. **Still build the case**: an unresolvable connector found in planning is not a reason to skip Phase 2. The stub is emitted, the structure lands, and the refusal happens at Check 14 — so the author gets a reviewable artifact plus the option to remove the rule and ship the remainder. Evidence table: [connector-trigger-impl.md § Placeholder fallback](connector-trigger-impl.md#placeholder-fallback). ## `tasks.md` Planning-Entry Shape From b5b214483361f146a1551cd413b8dddcbbeea7dd Mon Sep 17 00:00:00 2001 From: roberts-cliff Date: Thu, 13 Aug 2026 11:22:18 -0600 Subject: [PATCH 4/4] fix(maestro-case): gate Phase 5 on a non-empty issue log (Step 12.2 / Check 15) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `build-issues.md` was referenced in ~30 places across 10 files and verified in zero. Step 12's Checks 1-14 inspect caseplan.json and its sidecars, never the log; the Reporting clause *presumes* the file exists ("read the file after writing"); and the dump is a single terminal Write with no gate after it. The issue list is held in the agent's reasoning for the whole build (logging/impl-json.md § Setup) and dumped once at Step 12.1 -- so on a long build it is the first casualty of context pressure, and nothing notices. Observed 2026-08-13 on the claim-settlement e2e: 52 minutes, 8 stages, 39 tasks (every one a placeholder), 120 markers in tasks.md, NO build-issues.md at all -- and the task still scored 1.0, because no criterion looked. The operator receives 39 unwired resources with no record of which ones. Yesterday's bug-bash run of the same requirements DID write one, so this is nondeterministic compliance, not a missing feature. Same defect class as F2 (unenforced gate), F4 (nothing verifies what matters), and Check 14 before it was fixed: the contract says what to do, nothing checks it happened. Adds Step 12.2 as a Phase 4 exit gate -- it cannot join the Step 12 validator pass because the file does not exist until 12.1 has run. When the build carries unresolved work ( markers, placeholder tasks, or a surviving connector stub), build-issues.md must exist AND carry >= 1 entry; a header-only file fails. Repair is non-interactive: reconstruct from on-disk artifacts, stamp a NOTE marking severity/step attribution approximate, re-scan once, halt before Phase 5 if still empty. Test: _shared/check_build_issues_log.py + 14 pytest cases on synthetic stubs. Validated against both real runs -- fails this morning's 39-placeholder build, passes the connector-halt build. Mutation-tested (always-pass 7 fail, any-file-counts 3, ignore-placeholders 5, ignore-unresolved 1). Wired into the context_scale and connector-halt tasks; the context_scale criterion would have failed today's run. Its own test caught a bug in the grader: the logging template's `**Case file:** … | **Timestamp:** …` line fell through as an issue entry, so a header-only log passed. Fixed. Co-Authored-By: Claude Opus 5 --- skills/uipath-maestro-case/SKILL.md | 5 +- .../references/implementation.md | 28 ++- .../_shared/check_build_issues_log.py | 196 ++++++++++++++++++ .../_shared/test_check_build_issues_log.py | 175 ++++++++++++++++ .../context_scale/check_case_scale.py | 140 +++++++++++++ .../context_scale_claim_settlement.yaml | 123 +++++++++++ .../fixtures/claim-settlement.md | 155 ++++++++++++++ .../unresolved_connector_rule_halt.yaml | 8 +- 8 files changed, 824 insertions(+), 6 deletions(-) create mode 100644 tests/tasks/uipath-maestro-case/_shared/check_build_issues_log.py create mode 100644 tests/tasks/uipath-maestro-case/_shared/test_check_build_issues_log.py create mode 100644 tests/tasks/uipath-maestro-case/context_scale/check_case_scale.py create mode 100644 tests/tasks/uipath-maestro-case/context_scale/context_scale_claim_settlement.yaml create mode 100644 tests/tasks/uipath-maestro-case/context_scale/fixtures/claim-settlement.md diff --git a/skills/uipath-maestro-case/SKILL.md b/skills/uipath-maestro-case/SKILL.md index 3c099e1e9e..a6ea54802e 100644 --- a/skills/uipath-maestro-case/SKILL.md +++ b/skills/uipath-maestro-case/SKILL.md @@ -41,7 +41,7 @@ When `sdd.md` is absent, the case design belongs to **`uipath-planner` — the s 11. **Build-review preference decides the Phase 2 → Phase 3 boundary — captured ONCE, up front, never asked mid-build.** Capture it at journey start: greenfield-with-design-handoff folds it into the lane's Case Review Build options (`Build it — straight through` / `Build it — pause at the build preview` — Rule 1); greenfield-with-provided-SDD asks it once right after the roadmap; non-interactive runs and resumed runs with no recorded preference default to **straight-through** (no mid-build publish — Phase 5 stays the only publish point and stays gated). At the boundary, try `validate --skeleton-v2` first; fall back once to legacy `--skeleton` **only** when the parser response names `--skeleton-v2` as unknown or unsupported (typically `ErrorCode: invalid_argument` and exit 3). Exit 3 alone is not enough. A genuine validation failure means v2 ran — report it and do not hide it with a fallback. Name the selected profile in the counts summary; the gate is advisory and never halts on validation findings. Then: **straight-through** → continue into Phase 3 with no prompt, the summary line doubling as the milestone narration; **pause-at-preview** → follow the publish-for-review contract in [`references/phased-execution.md`](references/phased-execution.md) (AskUserQuestion `Publish for review` / `Skip publish and continue` / `Abort`; on publish, print `DesignerUrl` as plain text BEFORE the follow-up prompt — never only inside the question body). Hard stops that are NEVER bypassed regardless of preference: Phase 4 retry exhaustion (`Retry with fix` / `Pause for manual edit` / `Abort`), Phase 5 entry (`Publish to Studio Web` / `Skip to Debug`), Phase 6 entry (`Run debug session` / `Continue to publish`), and Phase 7 entry (`Publish to Orchestrator` / `Done`). Full contract in [`references/phased-execution.md`](references/phased-execution.md). 12. **Never run `uip maestro case debug` or the Phase 7 Orchestrator publish automatically.** `debug` executes the case for real — emails, messages, API calls. The Phase 7 chain (`uip solution pack` → `uip solution publish`) ships the case to the tenant solution feed. Both require explicit user consent via their own AskUserQuestion gate. 13. **All skill artifacts: Read + Write/Edit only.** Applies to `caseplan.json`, `sdd.md`, `sdd.draft.md`, `tasks.md`, `tasks/registry-resolved.json`, `tasks/trigger-spec-cache.json`, `tasks/spec-cache..json`, `bindings_v2.json`, `id-map.json`, `entry-points.json`, `build-issues.md`. No `python`, `node`, `jq`, `sed`, `awk`, or scripts that open/parse/modify/save these files. **Shell filesystem commands may not create, replace, rename, or relocate an artifact** — `cp`, `mv`, `install`, and `rsync` are forbidden for these files, including copying or renaming `sdd.draft.md` to `sdd.md`. **Specifically forbidden** (common slip): `node -e "...fs.writeFileSync..."`, `node -e "...fs.readFileSync..."`, `node -e "..." > `, `jq '...' > `, `python -c "...open(...,'w')..."`, `sed -i`, `awk -i inplace`, or any shell redirection (`>`, `>>`, `| tee`) onto a skill artifact regardless of interpreter. **Writing a helper script under `/tmp` or anywhere else to assemble a skill artifact is also forbidden** — the build-assembler pattern (`/tmp/build-caseplan.js`, `/tmp/gen-tasks.py`, etc.) is the same Rule 13 violation as inline `node -e`, regardless of "mechanical copy" or "avoid Read+Write churn" framing. If `caseplan.json` exceeds ~30KB and a single Write feels too large, split into the Phase-2-preview-then-Phase-3-detail cadence (per [case-editing-operations.md § Per-section batch write contract](references/case-editing-operations.md#per-section-batch-write-contract--canonical)) — never via helper script. **The `node -e ... fs.*` ban is not scoped to the artifact list — it applies to ALL file reads in this skill, including resource cache reads from `~/.uip/case-resources/`. Use `cat ... | python3 -c "..."` or the `Read` tool for cache lookups.** Bash subprocesses OK ONLY for UUID v4 generation (`node -e "console.log(crypto.randomUUID())"` for `operate.json.projectId` and `entry-points.json` `uniqueId` — subprocess MUST NOT `require('fs')` or use redirection), CLI metadata fetches, validate, debug, and solution scaffold/upload. **Prefixed IDs (`Stage_`, `t`, `Rule_`, etc.) are picked inline by the agent — no subprocess.** See [references/case-editing-operations.md § Tool usage](references/case-editing-operations.md#tool-usage--mandatory). -14. **Resolved resources must be runnable, and sidecar parity is an unconditional Phase 3 exit check.** Before Phase 4, run Step 12 Checks 7, 9, 11, 12, and 14 even when publish, debug, and `uip solution resources refresh` are skipped. A task with a non-null `selected` entry in `tasks/registry-resolved.json` MUST NOT be emitted as a placeholder: it must remain present with `data.name` and `data.folderPath` bound to complete root bindings, its resource must project into `bindings_v2.json.resources[]`, and its binding pair's `resourceKey` must be self-consistent with its own `name`/`folderPath` defaults (Check 11) — never a copied tenant identity/UUID. Checks 9 and 11 exempt connector nodes; **Check 12** covers them — a resolved connector task, event trigger, or `wait-for-connector` rule must carry the spliced `caseShape.context` plus its Connection/Folder root bindings, never the `typeId`+`connectionId`-only degraded shape. `uip maestro case validate` success does not substitute for these checks. On a mismatch, repair the named task/binding or regenerate the sidecar once as applicable, then re-check; halt before Phase 4 if any of these checks still fails. Repeat Check 7 before every `resources refresh`. Always run `resources refresh` before `uip solution upload` or `uip maestro case debug` so Studio Web can resolve dependencies. Every `uip solution upload` call MUST carry `--output-filter "{Status: Status, SolutionId: SolutionId, DesignerUrl: DesignerUrl}"` — the unfiltered response gets truncated and `DesignerUrl` is lost. +14. **Resolved resources must be runnable, and sidecar parity is an unconditional Phase 3 exit check.** Before Phase 4, run Step 12 Checks 7, 9, 11, 12, and 14 even when publish, debug, and `uip solution resources refresh` are skipped; before Phase 5, run the Step 12.2 issue-log gate (**Check 15**) — a build carrying `` markers or placeholder tasks MUST ship a non-empty `tasks/build-issues.md`, because the in-reasoning issue log is the first thing a long build loses and nothing else inspects it. A task with a non-null `selected` entry in `tasks/registry-resolved.json` MUST NOT be emitted as a placeholder: it must remain present with `data.name` and `data.folderPath` bound to complete root bindings, its resource must project into `bindings_v2.json.resources[]`, and its binding pair's `resourceKey` must be self-consistent with its own `name`/`folderPath` defaults (Check 11) — never a copied tenant identity/UUID. Checks 9 and 11 exempt connector nodes; **Check 12** covers them — a resolved connector task, event trigger, or `wait-for-connector` rule must carry the spliced `caseShape.context` plus its Connection/Folder root bindings, never the `typeId`+`connectionId`-only degraded shape. `uip maestro case validate` success does not substitute for these checks. On a mismatch, repair the named task/binding or regenerate the sidecar once as applicable, then re-check; halt before Phase 4 if any of these checks still fails. Repeat Check 7 before every `resources refresh`. Always run `resources refresh` before `uip solution upload` or `uip maestro case debug` so Studio Web can resolve dependencies. Every `uip solution upload` call MUST carry `--output-filter "{Status: Status, SolutionId: SolutionId, DesignerUrl: DesignerUrl}"` — the unfiltered response gets truncated and `DesignerUrl` is lost. 15. **Design handoff contract — invoke the `uipath-planner` Case Design Lane in THIS conversation; never a subagent.** When `sdd.md` is absent at the resolved path (or the user asks to finalize a case `sdd.draft.md`), load the `uipath-planner` skill and enter its Case Design Lane in **Build handoff** mode: the lane runs here, in the same context window, with the same user — the in-memory model, the resolution ledger, and the session's registry pull all cross the skill boundary for free. Hand off immediately on detection — before reading this skill's references, hunting the filesystem for templates, or running tenant commands (the lane owns tenant grounding). The lane owns everything through the Case Review's Build answer and the `sdd.md` write; this skill resumes at the first build action. User-facing language never mentions the handoff — the user sees one continuous flow. If `uipath-planner` is unavailable in this environment, say so in one line, ask the user to provide an `sdd.md` (or approve building from a design they paste), and stop — never improvise an interview, never fabricate an SDD. Cross-product planning beyond the case (multi-project solutions) stays a plain-text suggestion of `uipath-planner` — no handoff for that. **Receipt spot-check (one Grep — run it BEFORE Phase 1 reads any `sdd.md` this conversation did not watch being written, e.g. user-provided files):** the file must contain the four blueprint headings (`## Section 1: Case Definition` through `## Section 4: Integrations`) and at least one `##### Task` detail block. A freeform/summary SDD → do not build on it; route it back through the lane's template conformance gate. 16. **Caseplan task `type` enum is closed — 9 values, schema-kebab.** Any task node written into `caseplan.json` MUST have `type` exactly one of: `process` | `agent` | `rpa` | `action` | `api-workflow` | `case-management` | `execute-connector-activity` | `wait-for-connector` | `wait-for-timer`. **Never** write the plugin folder name (`connector-activity`, `connector-trigger`) or the CLI `--type` flag value into the JSON node — those name the planning artifacts, not the schema. Never write `external-agent`, `external-workflow`, `document-extraction`, `flow-process`, `wait-for-event`, or any hallucinated value — there is no plugin to back them. `external-agent`, `external-workflow`, `document-extraction`, and `flow-process` are **not supported yet**. See [references/case-schema.md § Task type](references/case-schema.md) and the Plugin Index naming-asymmetry table below. 17. **Empty registry lookup → AskUserQuestion BEFORE any placeholder fallback — unless the USER already ruled at the design-time gate.** When the same-session resolution ledger carries a `gateDecision` for the resource (`resolve-at-build` → placeholder; `create-during-build` → the inline-create flow below; `pick:` → bind that entry), execute it without re-asking — the gate ran once, at the planner's Case Review, and `gateDecision` exists only for items the user actually answered there. **An entry with NO `gateDecision` is a defaulted deferral** (no session, failed/pending pull, or non-interactive design) — it gets the full gate below, exactly like any fresh empty lookup; never treat a defaulted `resolve at build` as a user decision. Otherwise, when a planning-phase lookup returns 0 matches, present AskUserQuestion per lookup-batch (one prompt, not per-task) BEFORE any placeholder T-entry or per-plugin Unresolved Fallback, with options: (a) `Force pull and re-resolve` — loops back for still-empty; (b) `Use placeholders for all`; (c) `Create missing resources inline` — shown ONLY when ≥1 still-empty is creatable (an `agent` or an `api-workflow`) AND the CLI supports `registry --local`. **Create covers agents and API workflows only, gate-selected only** (never from SDD content alone; agent → `uipath-agents`, api-workflow → `uipath-api-workflow`); unselected + non-creatable empties (regular RPA process, action, case-management, connectors, agentic processes) → placeholder; the option is suppressed when `--local` is absent. Do NOT pre-judge via resource-name heuristics — the user's call. The gate and Select **group empties by `(name, type)`** (one row per resource, usages listed — non-creatable ones show only at the gate, annotated `placeholder only`); create-selected resources merge or split at [registry-discovery § 1c](references/registry-discovery.md#1c--dedup-the-selected-builds-one-resource-per-name-and-type) by I/O (identical-I/O usages → one build; differing → later renamed, anchor keeps the name; the SDD cell is updated only with user permission — never non-interactively). Placeholder fallback is valid only after `Use placeholders for all`. Build/register/verify mechanics live in [references/registry-discovery.md § Create-on-Missing](references/registry-discovery.md#create-on-missing-build-and-rediscovery) (gate detail: [§ MUST Confirm](references/registry-discovery.md#must-confirm-before-placeholder-fallback)). @@ -146,9 +146,10 @@ No hard stop on Phase 3 exit — proceed directly to Phase 4. ### Phase 4 — Validate -1. Run Step 12 once at the Phase 3 → Phase 4 boundary. It performs Checks 1–12, including bindings sidecar parity (Check 7), global output-ID uniqueness (Check 8), resolved-resource emission/preservation (Check 9), formal-arg slot ID format (Check 10), resourceKey self-consistency (Check 11), and connector node resolution completeness (Check 12). +1. Run Step 12 once at the Phase 3 → Phase 4 boundary. It performs Checks 1–14, including bindings sidecar parity (Check 7), global output-ID uniqueness (Check 8), resolved-resource emission/preservation (Check 9), formal-arg slot ID format (Check 10), resourceKey self-consistency (Check 11), connector node resolution completeness (Check 12), rule task-selector completeness (Check 13), and the case-fatal connector-stub halt (Check 14). 2. After all Step 12 checks pass, run full `uip maestro case validate`. Retry up to 3×; on 3rd failure **HARD STOP** AskUserQuestion: `Retry with fix` / `Pause for manual edit` / `Abort` 3. Dump `build-issues.md` (Step 12.1) +4. **Issue-log completeness gate (Step 12.2, Check 15)** — a build carrying `` markers, placeholder tasks, or a surviving connector stub MUST ship a non-empty `tasks/build-issues.md`. Reconstruct from on-disk artifacts and re-scan once; halt before Phase 5 if still absent or entry-less. ### Phase 5 — Publish diff --git a/skills/uipath-maestro-case/references/implementation.md b/skills/uipath-maestro-case/references/implementation.md index 1a48c8870e..14609ba69a 100644 --- a/skills/uipath-maestro-case/references/implementation.md +++ b/skills/uipath-maestro-case/references/implementation.md @@ -342,7 +342,33 @@ Run validate per [phased-execution.md § Phase 4](phased-execution.md#phase-4--v ## Step 12.1 — Dump issue log -Write issue list to `tasks/build-issues.md` per [`plugins/logging/impl-json.md`](plugins/logging/impl-json.md). On Phase 4 success → proceed to Phase 5. +Write issue list to `tasks/build-issues.md` per [`plugins/logging/impl-json.md`](plugins/logging/impl-json.md). + +## Step 12.2 — Issue-log completeness gate (Check 15) — PHASE 4 EXIT + +Runs immediately after the Step 12.1 dump; gates entry to Phase 5. Unlike Checks 1–14 this cannot live in the Step 12 validator pass — the file it inspects does not exist until 12.1 has run. + +**Predicate.** The build **carries unresolved work** when ANY of these hold: + +1. `tasks/tasks.md` contains one or more `` markers** emitted no `build-issues.md` at all and still passed every other gate. The operator was handed 39 unwired resources with no record of which ones. + +**Non-interactive repair.** Do not prompt. Reconstruct the log from the artifacts that survive on disk — `` markers in `tasks.md`, placeholder tasks in `caseplan.json`, unresolved entries in `tasks/registry-resolved.json` (`selected: null`), and any surviving connector stub — write `tasks/build-issues.md` per the logging plugin, then re-scan once. + +**A reconstructed log is explicitly second-best and must say so.** It records what the artifacts prove, not what the build actually observed: severities collapse to `SKIPPED`, and per-step provenance is gone. Stamp it with a literal line above the issue table: + +``` +NOTE: reconstructed at Step 12.2 from on-disk artifacts — the in-reasoning issue log was lost. Severity and step attribution are approximate. +``` + +Halt before Phase 5 if the file is still absent or entry-less after one repair pass. `uip maestro case validate` success does not satisfy this gate — a case with 39 unwired placeholders validates clean. + +On success → proceed to Phase 5. --- diff --git a/tests/tasks/uipath-maestro-case/_shared/check_build_issues_log.py b/tests/tasks/uipath-maestro-case/_shared/check_build_issues_log.py new file mode 100644 index 0000000000..0160708157 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/_shared/check_build_issues_log.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Grade Step 12.2 / Check 15 — a build carrying unresolved work must ship an issue log. + +Regression guarded +------------------ +The skill holds its issue list in the agent's reasoning for the whole build +(`plugins/logging/impl-json.md` § Setup) and dumps it exactly once at Step 12.1. +On a long build that in-memory list is the first casualty of context pressure, +and nothing downstream notices: Step 12 Checks 1-14 inspect `caseplan.json` and +its sidecars, never the log, and the Reporting clause *presumes* the file exists. + +Observed 2026-08-13: a 52-minute build produced 8 stages, 39 tasks -- every one +of them a placeholder -- and 120 `` markers in tasks.md, emitted NO +`build-issues.md` at all, and still scored 1.0. The operator was handed 39 +unwired resources with no record of which ones. + +Predicate +--------- +The build carries unresolved work when ANY of: + 1. tasks.md contains `= 1 issue entry. +A header-only or empty file fails: "I wrote a file" is not "I recorded the work". + +Exit 0 = pass, 1 = fail. Run from the sandbox root. +""" + +from __future__ import annotations + +import json +import re +import sys +from pathlib import Path + +SKIP_PARTS = {".venv", "node_modules", ".npm-prefix", "dist"} + + +def _iter(root: Path, name: str): + for p in root.rglob(name): + if SKIP_PARTS.isdisjoint(p.parts): + yield p + + +def find_one(root: Path, name: str) -> Path | None: + return next(_iter(root, name), None) + + +def walk_rules(case: dict): + """Every rule across all four condition scopes.""" + for cond in (case.get("metadata") or {}).get("caseExitRules") or []: + for grp in cond.get("rules") or []: + yield from grp + for node in case.get("nodes") or []: + if node.get("type") != "case-management:Stage": + continue + data = node.get("data") or {} + for key in ("entryConditions", "exitConditions"): + for cond in data.get(key) or []: + for grp in cond.get("rules") or []: + yield from grp + for lane in data.get("tasks") or []: + for task in lane: + for cond in task.get("entryConditions") or []: + for grp in cond.get("rules") or []: + yield from grp + + +def placeholder_tasks(case: dict) -> list[str]: + out = [] + for node in case.get("nodes") or []: + if node.get("type") != "case-management:Stage": + continue + for lane in (node.get("data") or {}).get("tasks") or []: + for task in lane: + if not (task.get("data") or {}): + out.append(task.get("displayName") or task.get("id") or "") + return out + + +def stub_rules(case: dict) -> int: + n = 0 + for rule in walk_rules(case): + if rule.get("rule") != "wait-for-connector": + continue + ctx = (rule.get("uipath") or {}).get("context") or [] + vals = {e.get("name"): e.get("value") for e in ctx if isinstance(e, dict)} + if vals.get("connectorKey") == "placeholder" or vals.get("operation") == "placeholder": + n += 1 + return n + + +def has_issue_entries(text: str) -> bool: + """A real entry, not just scaffolding. + + Accepts either the logging plugin's table rows or a bulleted/`##`-sectioned + log. Rejects headers, the reconstruction NOTE, separators, and blank lines -- + an empty template must not pass as a record. + """ + for raw in text.splitlines(): + line = raw.strip() + if not line or line.startswith("#"): + continue + if line.upper().startswith(("NOTE:", "BLOCKED:")): + continue + if set(line) <= set("|-: "): # markdown table separator + continue + # logging-plugin template scaffolding, e.g. + # **Case file:** caseplan.json | **Timestamp:** + if re.match(r"^\*\*(case file|timestamp)\s*:", line, re.IGNORECASE): + continue + if line.startswith("|"): + cells = [c.strip() for c in line.strip("|").split("|")] + if not cells or all(not c for c in cells): + continue + header_like = {"category", "severity", "step", "plugin", "message", + "errors", "warnings", "skipped", "issue", "detail"} + if all(c.lower() in header_like for c in cells if c): + continue + return True + if line.startswith(("- ", "* ", "1.")): + return True + return True + return False + + +def main() -> int: + root = Path.cwd() + + tasks_md = find_one(root, "tasks.md") + unresolved = 0 + if tasks_md is not None: + try: + unresolved = len(re.findall(r" 5 else "") + print( + "\nFAIL: the build carries unresolved work but shipped NO tasks/build-issues.md.\n" + f" {len(placeholders)} placeholder task(s){': ' + preview if preview else ''}\n" + f" {unresolved} marker(s) in tasks.md\n" + " The operator receives unwired resources with no record of which ones.\n" + " Step 12.2 requires a reconstructed log when the in-reasoning list was lost.\n" + " See implementation.md § Step 12.2 (Check 15)." + ) + return 1 + + if not has_issue_entries(text): + print( + f"\nFAIL: {issues.relative_to(root)} exists but carries no issue entries.\n" + " A header-only or empty log is not a record — Step 12.2 requires at least one entry\n" + " when the build carries unresolved work." + ) + return 1 + + reconstructed = "reconstructed at Step 12.2" in text + note = " (reconstructed log — severity/step attribution approximate)" if reconstructed else "" + print(f"\nPASS: unresolved work is recorded in {issues.relative_to(root)}.{note}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/tasks/uipath-maestro-case/_shared/test_check_build_issues_log.py b/tests/tasks/uipath-maestro-case/_shared/test_check_build_issues_log.py new file mode 100644 index 0000000000..a49f347d60 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/_shared/test_check_build_issues_log.py @@ -0,0 +1,175 @@ +"""Unit tests for check_build_issues_log.py — the Step 12.2 / Check 15 grader. + +Run with ``pytest``; CI runs them via ``pytest tests/tasks/uipath-maestro-case/``. + +Fixtures are minimal synthetic stubs built inline — no tenant, no captured +artifact. The grader was additionally validated against two real runs: +a 39-placeholder build with no log (fails) and the connector-halt build (passes). +""" + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +CHECKER = Path(__file__).with_name("check_build_issues_log.py") + +STUB_CTX = [ + {"name": "connectorKey", "value": "placeholder", "type": "string"}, + {"name": "operation", "value": "placeholder", "type": "string"}, +] + +REAL_LOG = """# Build Issues — Stub + +| Category | Errors | Warnings | Skipped | +|---|---|---|---| +| io-binding | 0 | 0 | 2 | + +## io-binding + +| Severity | Step | Message | +|---|---|---| +| SKIPPED | 9 | Task "Do Thing" unresolved — placeholder emitted | +""" + +HEADER_ONLY_LOG = """# Build Issues — Stub + +**Case file:** caseplan.json | **Timestamp:** 2026-08-13T00:00:00Z + +| Category | Errors | Warnings | Skipped | +|---|---|---|---| + +## Open Items for User +""" + + +def caseplan(*, placeholder=False, stub=False): + task = {"id": "tAAAAAAAA", "type": "wait-for-timer", "displayName": "Hold"} + task["data"] = {} if placeholder else {"serviceType": "X"} + stage = { + "id": "Stage_aaaaaa", + "type": "case-management:Stage", + "data": {"label": "S", "tasks": [[task]]}, + } + if stub: + stage["data"]["entryConditions"] = [{ + "id": "Condition_a", "displayName": "E", + "rules": [[{"id": "Rule_a", "rule": "wait-for-connector", + "uipath": {"context": STUB_CTX, "inputs": [], "outputs": []}}]], + }] + return {"id": "case-Stub000001", "version": "27.0.0", "name": "Stub", + "metadata": {}, "nodes": [stage], "edges": []} + + +def run(tmp_path, *, case=None, tasks_md=None, log=None): + if case is not None: + proj = tmp_path / "Sol" / "Proj" + proj.mkdir(parents=True, exist_ok=True) + (proj / "caseplan.json").write_text(json.dumps(case)) + if tasks_md is not None: + d = tmp_path / "tasks" + d.mkdir(parents=True, exist_ok=True) + (d / "tasks.md").write_text(tasks_md) + if log is not None: + d = tmp_path / "tasks" + d.mkdir(parents=True, exist_ok=True) + (d / "build-issues.md").write_text(log) + return subprocess.run([sys.executable, str(CHECKER)], cwd=tmp_path, + capture_output=True, text=True) + + +# --- the regression this exists for ------------------------------------------------- + +def test_placeholders_without_log_fails(tmp_path): + """Today's 39-placeholder build: unresolved work, no log at all.""" + res = run(tmp_path, case=caseplan(placeholder=True)) + assert res.returncode == 1, res.stdout + assert "shipped NO tasks/build-issues.md" in res.stdout + + +def test_unresolved_markers_without_log_fails(tmp_path): + res = run(tmp_path, case=caseplan(), tasks_md="## T01\n- id: \n") + assert res.returncode == 1, res.stdout + assert "1 marker" in res.stdout + + +def test_surviving_stub_without_log_fails(tmp_path): + res = run(tmp_path, case=caseplan(stub=True)) + assert res.returncode == 1, res.stdout + + +def test_placeholders_with_real_log_passes(tmp_path): + res = run(tmp_path, case=caseplan(placeholder=True), log=REAL_LOG) + assert res.returncode == 0, res.stdout + assert "PASS" in res.stdout + + +# --- "I wrote a file" is not "I recorded the work" ---------------------------------- + +def test_header_only_log_fails(tmp_path): + """An empty template must not pass as a record.""" + res = run(tmp_path, case=caseplan(placeholder=True), log=HEADER_ONLY_LOG) + assert res.returncode == 1, res.stdout + assert "no issue entries" in res.stdout + + +def test_empty_log_fails(tmp_path): + res = run(tmp_path, case=caseplan(placeholder=True), log="") + assert res.returncode == 1, res.stdout + + +def test_note_line_alone_is_not_an_entry(tmp_path): + log = "# Build Issues\n\nNOTE: reconstructed at Step 12.2 from on-disk artifacts\n" + res = run(tmp_path, case=caseplan(placeholder=True), log=log) + assert res.returncode == 1, res.stdout + + +# --- clean builds don't need a log --------------------------------------------------- + +def test_clean_build_without_log_passes(tmp_path): + res = run(tmp_path, case=caseplan(), tasks_md="## T01\n- id: real-id\n") + assert res.returncode == 0, res.stdout + assert "no unresolved work" in res.stdout + + +# --- accepted log shapes ------------------------------------------------------------- + +@pytest.mark.parametrize("body", [ + "# Build Issues\n\n- SKIPPED: task X unresolved\n", + "# Build Issues\n\n## io-binding\n\n| Severity | Step | Message |\n|---|---|---|\n| SKIPPED | 9 | x |\n", + "# Build Issues\n\nTask X was left unresolved and needs wiring.\n", +]) +def test_various_real_log_shapes_pass(tmp_path, body): + res = run(tmp_path, case=caseplan(placeholder=True), log=body) + assert res.returncode == 0, res.stdout + + +def test_reconstructed_log_is_flagged_in_output(tmp_path): + log = ("# Build Issues\n\nNOTE: reconstructed at Step 12.2 from on-disk artifacts — " + "the in-reasoning issue log was lost.\n\n- SKIPPED: task X unresolved\n") + res = run(tmp_path, case=caseplan(placeholder=True), log=log) + assert res.returncode == 0, res.stdout + assert "reconstructed log" in res.stdout + + +# --- environment guards --------------------------------------------------------------- + +def test_venv_copies_ignored(tmp_path): + venv = tmp_path / ".venv" / "sample" + venv.mkdir(parents=True) + (venv / "caseplan.json").write_text(json.dumps(caseplan(placeholder=True))) + res = run(tmp_path, case=caseplan(), tasks_md="## T01\n- id: real\n") + assert res.returncode == 0, res.stdout + + +def test_unparseable_caseplan_fails(tmp_path): + proj = tmp_path / "Sol" / "Proj" + proj.mkdir(parents=True) + (proj / "caseplan.json").write_text("{ not json") + res = subprocess.run([sys.executable, str(CHECKER)], cwd=tmp_path, + capture_output=True, text=True) + assert res.returncode == 1 + assert "unparseable" in res.stdout diff --git a/tests/tasks/uipath-maestro-case/context_scale/check_case_scale.py b/tests/tasks/uipath-maestro-case/context_scale/check_case_scale.py new file mode 100644 index 0000000000..d05619f792 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/context_scale/check_case_scale.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +"""Scale-and-shape guard for the context_scale fixture. + +This fixture exists to measure *cost* (tokens, wall clock, peak context) across +skill variants. Cost is only comparable if every variant built a case of the +same magnitude — otherwise a variant "wins" by building less. This checker is +the Goodhart guard: it asserts the case is genuinely large and carries the +shapes the requirements call for, without pinning exact names (a variant may +name stages differently and still be correct). + +Floors are set from the 2026-08-11 measured baseline, which produced: + stages=9 tasks=33 conditions=66 stage_slas=5 + secondary=4 return_to_origin=2 case_exits=3 +Each floor sits below the baseline with headroom, so a faithful build passes +and a truncated one does not. + +Schema notes (verified against a real caseplan, schema v27): + - stage.data.tasks is a list of task *sets*; each set is a list of tasks. + - stage SLA lives at stage.data.slaRules (absent on secondary stages). + - a secondary stage carries stage.data.stageType; primaries do not. + - case-level exits live at metadata.caseExitRules, not at the root. + +Exit 0 = pass. Exit 1 = fail, with every unmet expectation listed. +""" +import json +import os +import sys + +CANDIDATES = [ + "AutoClaimSettlement/AutoClaimSettlementCase/caseplan.json", + "AutoClaimSettlement/AutoClaimSettlement/caseplan.json", +] + +MIN_STAGES = 7 +MIN_TASKS = 24 +MIN_CONDITIONS = 45 +MIN_STAGE_SLAS = 3 +MIN_CASE_EXITS = 3 +STAGE_TYPE = "case-management:Stage" + + +def find_caseplan(): + for rel in CANDIDATES: + if os.path.isfile(rel): + return rel + for root, dirs, files in os.walk("."): + dirs[:] = [d for d in dirs if d not in (".git", "node_modules", ".venv", "__pycache__")] + if "caseplan.json" in files: + return os.path.join(root, "caseplan.json") + return None + + +def flatten_task_sets(raw): + """stage.data.tasks is a list of task sets; tolerate a flat list too.""" + out = [] + for entry in raw or []: + if isinstance(entry, list): + out.extend(x for x in entry if isinstance(x, dict)) + elif isinstance(entry, dict): + out.append(entry) + return out + + +def main(): + path = find_caseplan() + if not path: + print("FAIL: no caseplan.json found under the working directory") + return 1 + + with open(path) as fh: + plan = json.load(fh) + + metadata = plan.get("metadata") or {} + stages = [n for n in (plan.get("nodes") or []) if n.get("type") == STAGE_TYPE] + + tasks = [] + stage_slas = secondary = return_to_origin = conditions = 0 + + for st in stages: + data = st.get("data") or {} + stage_tasks = flatten_task_sets(data.get("tasks")) + tasks.extend(stage_tasks) + + if data.get("slaRules"): + stage_slas += 1 + if data.get("stageType"): + secondary += 1 + + entry = data.get("entryConditions") or [] + exit_ = data.get("exitConditions") or [] + conditions += len(entry) + len(exit_) + for t in stage_tasks: + conditions += len(t.get("entryConditions") or []) + + for cond in exit_: + blob = json.dumps(cond) + if "return-to-origin" in blob or "returnToOrigin" in blob: + return_to_origin += 1 + + case_exits = len(metadata.get("caseExitRules") or []) + case_sla = bool(metadata.get("slaRules")) + + failures = [] + if len(stages) < MIN_STAGES: + failures.append(f"stages: {len(stages)} < {MIN_STAGES}") + if len(tasks) < MIN_TASKS: + failures.append(f"tasks: {len(tasks)} < {MIN_TASKS}") + if conditions < MIN_CONDITIONS: + failures.append(f"conditions: {conditions} < {MIN_CONDITIONS}") + if stage_slas < MIN_STAGE_SLAS: + failures.append(f"stages carrying an SLA: {stage_slas} < {MIN_STAGE_SLAS}") + if not case_sla: + failures.append("no case-level SLA (metadata.slaRules) — requirements set a 10-day target") + if secondary < 1: + failures.append("no secondary stage — requirements describe turn-down, withdrawal, " + "and escalation lanes") + if return_to_origin < 1: + failures.append("no return-to-origin stage exit — escalation must resume the " + "interrupted stage") + if case_exits < MIN_CASE_EXITS: + failures.append(f"case exit routes: {case_exits} < {MIN_CASE_EXITS} " + "(paid / turned down / withdrawn)") + + print(f"caseplan: {path}") + print(f" stages={len(stages)} tasks={len(tasks)} conditions={conditions} " + f"stage_slas={stage_slas} case_sla={case_sla} secondary={secondary} " + f"return_to_origin={return_to_origin} case_exits={case_exits}") + + if failures: + print("FAIL — build is smaller or simpler than the requirements demand:") + for f in failures: + print(f" - {f}") + return 1 + + print("PASS — case is at comparable scale to the measured baseline") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/tasks/uipath-maestro-case/context_scale/context_scale_claim_settlement.yaml b/tests/tasks/uipath-maestro-case/context_scale/context_scale_claim_settlement.yaml new file mode 100644 index 0000000000..7bac741e65 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/context_scale/context_scale_claim_settlement.yaml @@ -0,0 +1,123 @@ +task_id: skill-case-context-scale-claim-settlement +description: > + COST FIXTURE for skill-variant A/B comparison (see + tests/experiments/skill-comparison-playbook.md). Builds a large auto-insurance + claim-settlement case from staged business requirements — the same input as the + measured 2026-08-11 baseline run, which produced 9 stages / 33 tasks / 63 + conditions in 79.9 min, 986k output tokens, and 903.6k peak context. + + The graded outcome is deliberately modest: a case that validates and is at + comparable scale. The point is not pass/fail — it is the `total_tokens` and + `average_duration` that `variant.json` records alongside it. Smoke-tier case + tasks are too small to separate context-management strategies; this one is + sized so the difference is measurable. + + check_case_scale.py is the Goodhart guard: without a scale floor, a variant + could post a cheap run by building a smaller case. Floors sit below the + measured baseline with headroom and pin shapes, not names. + + Offline resources are expected. The eval tenant has no claim connectors, + action app, or child case, so placeholders are correct here and + are NOT graded as defects. +tags: [uipath-maestro-case, e2e, mode:build, lifecycle:generate, shape:multi-node, feature:approval-gate, feature:escalation] + +# Sized from the measured baseline (79.9 min end-to-end) plus room for one cold +# restart after an API-limit death, matching the golden_rebuild precedent. +run_limits: + task_timeout: 14400 + max_turns: 600 + turn_timeout: 9000 + +# `Agent` must stay in allowed_tools. The baseline run had it available and +# never used it — it chained four in-process `Skill` calls instead — and the +# whole point of the comparison is whether a subagent-scoped variant behaves +# differently. Removing Agent would make the variant under test unrunnable and +# silently convert this into a different experiment. Extends the experiment's +# allowed_tools; keep the base list in sync with experiments/default.yaml. +agent: + allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep", "Agent"] + +sandbox: + template_sources: + - type: template_dir + path: fixtures + +initial_prompt: | + Build a UiPath Case Management solution named "AutoClaimSettlement" containing + a case project named "AutoClaimSettlementCase", from the business requirements + staged as `claim-settlement.md` in this folder. + + The requirements are business prose, not an SDD. Design the case from them and + cover every path they describe, including the ones off the happy path. + + Important: + - The `uip` CLI is available and already authenticated. + - This tenant hosts none of the referenced external systems. Unresolved + resource placeholders are expected and acceptable; do not invent identifiers + to fill them. + - Treat every hard stop after the design as pre-approved — planning approval, + build-review preview, and publish gates. Choose the continue / skip-publish + path and keep going. + - Do NOT call AskUserQuestion; it is not available in this harness. Decide by + best assumption and record the assumption in the design document. + - Do NOT run `uip maestro case debug` and do NOT publish the solution. + - Do NOT ask for approval, confirmation, or feedback, and do NOT pause between + planning and implementation. + - Before starting, load the uipath-maestro-case skill and follow its workflow. + + Not complete until + `uip maestro case validate AutoClaimSettlement/AutoClaimSettlementCase/caseplan.json --output json` + passes. + +success_criteria: + - type: run_command + description: "uip maestro case validate passes on the generated caseplan.json" + command: "uip maestro case validate AutoClaimSettlement/AutoClaimSettlementCase/caseplan.json --output json" + timeout: 120 + expected_exit_code: 0 + weight: 3.0 + pass_threshold: 1.0 + + - type: run_command + description: "case is at comparable scale to the measured baseline: stage/task/condition floors, a case SLA, a secondary lane, a return-to-origin exit, and three case exit routes" + command: "python3 $TASK_DIR/check_case_scale.py" + timeout: 120 + expected_exit_code: 0 + weight: 3.0 + pass_threshold: 1.0 + + - type: run_command + description: > + Step 12.2 / Check 15 — this tenant hosts none of the referenced systems, so the + build is expected to be placeholder-heavy. A build carrying unresolved work MUST + ship a non-empty tasks/build-issues.md; otherwise the operator receives unwired + resources with no record of which ones. + command: "python3 $SKILLS_REPO_PATH/tests/tasks/uipath-maestro-case/_shared/check_build_issues_log.py" + timeout: 30 + expected_exit_code: 0 + weight: 2.0 + pass_threshold: 1.0 + + - type: file_exists + description: "Design document was produced as a reference artifact" + path: "sdd.md" + weight: 1.5 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Agent did not use mutating case-authoring CLI commands" + tool_name: "Bash" + command_pattern: '(uip|\$UIP)\s+maestro\s+case\s+(cases|stages|tasks|(?:case|stage|task)-(?:entry|exit)-conditions)\s+(add(?:-connector)?|update|remove)\b' + weight: 3.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Agent did not run case debug or publish (debug executes for real; publish is out of scope)" + tool_name: "Bash" + command_pattern: '(uip|\$UIP)\s+maestro\s+case\s+(debug|publish)\b' + weight: 1.5 + pass_threshold: 1.0 + +post_run: + - command: "find . -maxdepth 5 -type d \\( -name node_modules -o -name .npm-prefix -o -name .venv \\) -prune -exec rm -rf {} +" + timeout: 60 diff --git a/tests/tasks/uipath-maestro-case/context_scale/fixtures/claim-settlement.md b/tests/tasks/uipath-maestro-case/context_scale/fixtures/claim-settlement.md new file mode 100644 index 0000000000..05aa48ec72 --- /dev/null +++ b/tests/tasks/uipath-maestro-case/context_scale/fixtures/claim-settlement.md @@ -0,0 +1,155 @@ +# Auto Insurance Claim Settlement — Process Requirements + +## What this process is + +When a policyholder's car is damaged, they file a claim with their insurance company. This process handles that claim from the moment it's submitted until it's fully resolved — the claimant gets paid, the claim is turned down, or the claimant decides to withdraw it. + +Every claim gets its own reference number (e.g. `CLM-001`, `CLM-002`) so everyone involved can track it. + +## How a claim begins + +A claim starts when a policyholder submits one. Along with the claim, they provide: + +- Their name and email +- Their policy number +- What kind of damage it is (Collision, Theft, Weather, Vandalism, or Other) +- How much they're claiming, and in what currency +- A written description of what happened +- Photos and documents that support the claim (damage photos, repair estimates, police reports) +- The date they submitted it + +At any point while the claim is open, the claimant and the people handling it can add more documents and leave comments on the claim. + +## How fast things should move + +A claim should be fully resolved within **10 business days** of being filed. Each phase of the process has its own target too: + +| Phase | Target | +|---|---| +| Checking the claim | 1 business day | +| Adjuster review | 3 business days | +| Senior review | 3 business days | +| Paying the claimant | 2 business days | +| Wrapping up (paid, turned down, or withdrawn) | 1 business day | + +If a phase is running out of time — say it's used up 70% of its target — the people handling the claim should see a warning so they can act before the deadline passes. + +If a phase misses its deadline entirely, the process doesn't just show it — it acts on its own: + +- The claim is escalated: the claims team lead gets a task to step in and unblock it. +- The claimant gets a status update apologizing for the delay and giving a new expected date. + +And if the claim misses its overall 10-day target, the team lead also gets a service-recovery task to review what went wrong before the claim closes. + +## How a claim moves through the process + +### 1. Checking the claim + +When a claim first arrives, it gets a once-over before anyone spends time reviewing it: + +- The claim details are checked: Is the policy active? Is the amount within what the policy covers? Is anything missing? Are photos or documents attached? If something's wrong, the problems are noted so they can be fixed. +- The policyholder's records are pulled up: their policy details, coverage limits, past claims, and which adjuster handles their account. +- The description and photos are read to confirm the type of damage matches what the claimant selected — and to flag anything that looks odd. (This is a helpful extra, not a must.) + +The claimant can add more photos or documents at any time during this phase. + +Once everything checks out, the claim moves to the adjuster. If the claimant changes their mind and withdraws, the claim skips ahead to the withdrawal wrap-up. + +One more thing: if the adjuster later finds problems and sends the claim back for corrections (see next phase), the claim comes **back to this phase** and goes through these checks again. + +### 2. Adjuster review + +The claim now goes to the adjuster assigned to this policyholder: + +- The adjuster gets an email with a summary of the claim and links to approve or decline it. +- The claim then waits for the adjuster to respond. +- The adjuster looks over the full claim, adds their comments, and makes a call: approve it, decline it, or send it back to the claimant to fix something. +- If the adjuster sits on it too long, the claim automatically gets bumped up to a senior adjuster so it doesn't stall. +- While reviewing, the adjuster can also email the claimant to ask for clarification or missing paperwork — whenever they need to. + +From here, the claim can go four ways: + +- **Adjuster approves** → the claim moves on to senior review. +- **Adjuster declines** → the claim goes to the turn-down wrap-up. +- **Adjuster sends it back for corrections** → the claim returns to "Checking the claim." +- **Claimant withdraws** → the claim goes to the withdrawal wrap-up. + +### 3. Senior review + +Before any money goes out, the claim gets a deeper look: + +- The claim is checked against company policy: Has this been submitted before? Is there enough coverage left on the policy? Does it comply with the policy's terms? This produces a risk score and flags anything of concern. +- The claim is compared against past claim patterns to spot warning signs of fraud — for example, several recent claims on the same policy, amounts that sit just under approval limits, or unusual timing. (This is a helpful extra, not a must.) +- The company confirms funds are available, and the right level of sign-off is determined by the amount: claims under $500 are approved automatically, claims under $5,000 only need the adjuster's earlier approval, and claims of $5,000 or more need a senior reviewer's sign-off. +- A senior reviewer then sees the whole picture — the claim, the compliance results, and the risk score — and makes the final decision. **The claim waits for them to explicitly choose what happens next**: send it to payment, or turn it down. It never moves forward on its own from here. + +If the reviewer turns it down, the claim goes to the turn-down wrap-up. The claimant can still withdraw during this phase too. + +### 4. Paying the claimant + +Once a senior reviewer sends the claim to payment: + +- A payment for the approved amount is set up and sent to the claimant's bank account. +- Optionally, a separate tracking record is opened just for following the payout, linked back to the claim, so the payment can be followed on its own. +- The claim waits for the bank to confirm the money went through, then the payment is marked complete. + +If the payment goes through, the claim moves to the "paid" wrap-up. If the payment fails — wrong bank details, or a fraud hold — the claim goes to the turn-down wrap-up instead. + +### 5. Wrap-up: Claim paid + +The happy ending. Two final things happen, and then the claim is closed: + +- The claimant gets a confirmation message with the amount, a payment reference, and when to expect the deposit. +- The payout is recorded in the company's books against the policy. + +Once a claim reaches this point, it's done — it can't move anywhere else. + +### 6. Wrap-up: Claim turned down + +A claim can be turned down by the adjuster, by the senior reviewer, or because payment failed. When that happens: + +- The claimant gets an email explaining why it was turned down, who made the decision, and how to resubmit or appeal. +- The decision is logged for the company's audit records — what was turned down, by whom, why, and when. + +Then the claim is closed. It can't move anywhere else. + +### 7. Wrap-up: Claim withdrawn + +The claimant can withdraw their claim at any point before payment — while it's being checked, while the adjuster has it, or while it's in senior review. When they do: + +- The claimant gets an email confirming the withdrawal. +- Anything still in motion is tidied up: pending reviews are cancelled, countdown timers are switched off, and the claim is marked withdrawn. + +Then the claim is closed. It can't move anywhere else. + +## Actions people can take at any time + +Beyond the sequenced steps above, the people on a claim must be able to take these actions whenever they judge it necessary, at any point while the claim is in one of the three active phases: + +- **Add to the claim** — the claimant and handlers can attach documents and leave comments. +- **Ask the claimant a question** — any handler can email the claimant for clarification or missing paperwork. +- **Order a second opinion** — any handler who wants extra certainty can order an independent damage inspection from someone not already involved in the claim; the findings are added to the claim for the next decision-maker to see. +- **Pull in a manager** — even when the amount doesn't require it, a handler who is uneasy about a claim can ask a manager to look at it and give a sign-off; the claim waits for that opinion before moving on. +- **Withdraw** — the claimant can withdraw the claim (as described in the phases above). + +## Things this process must handle + +- **Going backwards** — the adjuster can send a claim back for corrections, and it goes through the initial checks again before returning to review. +- **Withdrawing at any time** — the claimant can pull out during any of the three active phases, and everything wraps up cleanly. +- **No stalling** — a review that sits untouched gets escalated automatically. +- **Deadlines that act** — reaching 70% of a target warns people; missing the deadline triggers real work on its own (an escalation task for the team lead, a delay notice to the claimant), not just a visual change. +- **A person decides the path** — after senior review, the claim never advances by itself; it waits for the reviewer's explicit choice. +- **Side-by-side tracking** — the payout can be followed through its own linked tracking record without holding up or cluttering the main claim. +- **On-demand actions** — the actions under "Actions people can take at any time" aren't steps in a sequence; a handler can order a second opinion or pull in a manager whenever their judgment says so, and the claim waits where it needs to. + +## When we'd call this done + +The process should be able to carry a claim through every path end to end: + +- A claim that gets paid +- A claim turned down by the adjuster, one turned down by the senior reviewer, and one turned down because the payment failed +- A claim the claimant withdraws +- A claim sent back for corrections, fixed, and resubmitted through to a decision + +And once a claim is closed — paid, turned down, or withdrawn — nothing can reopen or move it. + diff --git a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml index cb07da4e24..a21decfe7b 100644 --- a/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml +++ b/tests/tasks/uipath-maestro-case/unresolved_connector_rule_halt/unresolved_connector_rule_halt.yaml @@ -96,9 +96,11 @@ success_criteria: pass_threshold: 1.0 - type: run_command - description: "The build recorded its issues rather than exiting silently" - command: "find . -name build-issues.md -not -path '*/.venv/*' | grep -q ." - timeout: 10 + description: > + Step 12.2 / Check 15 — the build recorded its unresolved work rather than exiting + silently. Stronger than a file_exists: a header-only log fails. + command: "python3 $SKILLS_REPO_PATH/tests/tasks/uipath-maestro-case/_shared/check_build_issues_log.py" + timeout: 30 expected_exit_code: 0 weight: 1.0 pass_threshold: 1.0